SentinelOne Endpoint Security conflict with Headless Mode - E_FAIL (0x80004005) exit code -107374181

Discussions related to using VirtualBox on Windows hosts.
Post Reply
fstephane
Posts: 14
Joined: 24. Sep 2020, 22:58

SentinelOne Endpoint Security conflict with Headless Mode - E_FAIL (0x80004005) exit code -107374181

Post by fstephane »

The company I work for distributes a local web server that runs in VMs, some of which are hosted on locally installed VirtualBox instances on client systems.

Recently, we've had three clients report that the VM is not working. All three client VMs have been working without issue for at least several months now. All three have SentinelOne Endpoint Security installed on the host system.
The VMs are failing when run using Headless Start, but work fine (as far as I can tell) on Normal Start. The error we're getting in each case is:
E_FAIL.png
E_FAIL.png (61.4 KiB) Viewed 2918 times
Failed to open a session for the virtual machine <name>.
The virtual machine '<name>' has terminated unexpectedly during startup with exit code -1073741819
E_FAIL (0x80004005)
The VMs only produce Hardening Logs when run on Normal Start, but not Headless. I gather this means some other software (I suspect SentinelOne) is interfering with the Headless EXE before it has a chance to write to the log. I have attached a version of the logs folder for both Headless and Normal. In a couple of the cases, IT has confirmed that SentinelOne did not report blocking anything to do with VirtualBox. But I think some people on this forum have suggested that - due to the way VBox Hardening works - that doesn't necessarily mean SentinelOne isn't the culprit.

I'm thinking there must have been an update, either to Windows or SentinelOne, that might explain why these clients started failing at the same time. There were no updates to their VirtualBox versions or our VM around the time of the reported issues.

We have tried VBox 6.1.26 and 6.1.34. We also tried VBox 5, but in every case the Manager for version 5 wouldn't even start and gave us an error in the event viewer (apologies I don't have a screenshot of that).

I don't think this gives much more detail, but here's the event viewer error we get for Headless start in VBox 6:
Faulting application name: VBoxHeadless.exe, version: 6.1.34.636, time stamp: 0x623a5dfe
Faulting module name: VBoxHeadless.exe, version: 6.1.34.636, time stamp: 0x623a5dfe
Exception code: 0xc0000005
Fault offset: 0x0000000000014e0b
Faulting process id: 0xc28
Faulting application start time: 0x01d881c8945baeb8
Faulting application path: C:\Program Files\Oracle\VirtualBox\VBoxHeadless.exe
Faulting module path: C:\Program Files\Oracle\VirtualBox\VBoxHeadless.exe
Report Id: 5ffd9d54-b636-45e1-916b-319dc1b60a85
Faulting package full name:
Faulting package-relative application ID:
Interestingly, in one case the client IT was willing to temporarily uninstall SentinelOne and restart the host computer. When we tried running the VM in Headless, it gave us a different error. I'm at the attachment limit on this post but I'll copy the description here:
Failed to open a session for the virtual machine
The VM session was closed before any attempt to power it on
E_FAIL (0x80004005)
After getting that new error, we re-imported the VM in VBox and then it worked! The IT even re-installed SentinelOne, rebooted the computer - and we were still able to start from Headless. I don't know if there were other factors at play, but it seems like the temporary uninstall of SentinelOne might have had something to do with it.

In both the other cases, I've requested a temporary uninstall of SentinelOne, but IT has all but refused to do so. Someone on this post (viewtopic.php?f=6&t=104692) suggested some possible exceptions to try, so I can run that by IT, but I'm not sure how likely that is to work or for them to even try it.

I'm wondering if anyone else has had recent issues with SentinelOne that match my issue? Also wondering if there's any new insight on how to get VBox and Sentinel to co-exist. I've been through every post I can find on this forum but so far the only reliable solution seems to be to uninstall SentinelOne completely, which I don't think will pan out in our case unfortunately.

Is this something that Oracle might be looking into resolving in a future patch?

Any info or help would be greatly appreciated!
Attachments
NormalLogs.zip
(158.28 KiB) Downloaded 5 times
HeadlessLogs.zip
(34.98 KiB) Downloaded 6 times
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: SentinelOne Endpoint Security conflict with Headless Mode - E_FAIL (0x80004005) exit code -10737

Post by scottgus1 »

00:00:04.488460 OS Product: Windows 2016
00:00:04.488461 OS Release: 10.0.14393
00:00:04.488462 OS Service Pack:
00:00:04.620394 DMI Product Name: VMware Virtual Platform
Looks like your workstation is a VM itself, and your company is trying interspecies nested virtualization. Typically not a supported setup, but let's take a whack at it anyway.

Both the normal and the headless logs seem to show working VM 'hardware', a booting and running VM OS, and successfully-starting Guest additions. No mention of an error. This could be the case if the VM is not starting up enough to start a new log, which is supported by the dates of the logs, being from the 9th and 13th of June 2022, and today is June 27, '22. So the logs don't seem to show the error condition.

AV and other malware companies do put out updates that sometimes interfere with Virtualbox. SentinelOne is one of them.

There was a bug or such over headless start some short time ago, but I can't find mention of it at the moment. But if the update logs for SentinelOne coordinate with the time that Virtualbox started to have trouble, then the SentinelOne devs might want to contact the Virtualbox devs on IRC.
fstephane wrote:I've requested a temporary uninstall of SentinelOne, but IT has all but refused to do so.
What does your boss say? If the boss is sold on Virtualbox running under VMware with SentinelOne, then the boss will make IT do what's necessary. Gotta sell the boss on this.

One other thing, if Virtualbox is going to remain in the mix per the boss's requirements, then the company might want to pony up for a support contract, see oracle store. Probably not too expensive for a company, lets you get more direct support than these user forums, and let's you legally use the Extension Pack too.
fstephane
Posts: 14
Joined: 24. Sep 2020, 22:58

Re: SentinelOne Endpoint Security conflict with Headless Mode - E_FAIL (0x80004005) exit code -10737

Post by fstephane »

Thanks for the response,
Looks like your workstation is a VM itself, and your company is trying interspecies nested virtualization
Yes we did not realize the host was a VM until the issue started occurring. I'm not sure if this is the case with the other two clients. We recommend against this in our requirements, but in this case it was working for over a year before Headless started failing. Along with the timing of issues with other clients running SentinelOne, it appears the problem has more to do with AV than nested virtualization.

[Update] I've confirmed at least one of the machines is not a VM

The timestamp on the log files don't appear to update when we run Headless so I think you're right that it is not writing to the logs at all.

I can try to confirm if there's been a coincident SentinelOne update, and contact the devs to try and get them talking to the VBox devs.

It wouldn't be my boss' call in this case. The client hires the IT firm, so we'd need to suggest that they discuss possibilities with their own IT. Sometimes that works so I'll give it a try. I'm not sure how many of our clients are running SentinelOne on the VM hosts so I'd prefer a solution that IT would be more open to. But if that's the only option at this point then it is what it is
fstephane
Posts: 14
Joined: 24. Sep 2020, 22:58

Re: SentinelOne Endpoint Security conflict with Headless Mode - E_FAIL (0x80004005) exit code -10737

Post by fstephane »

We were able to resolve the issue for one client by getting their IT to add the following exceptions to SentinelOne (based on a suggestion in this post: viewtopic.php?f=6&t=104692):

  • - C:\Program Files\Oracle
    - C:\Users\<user_account>\.VirtualBox
    - C:\Window\System32\config\systemprofile\.VirtualBox (we copy the ".VirtualBox" folder here in order to start up headless from a "Local System Account" Windows service without logging in as a user)
    - <location of the VM files>
    - C:\Windows\System32\drivers\VBoxSup.sys
    - C:\Windows\System32\drivers\VBoxNetLwf.sys
    - C:\Windows\System32\drivers\VBoxNetAdp6.sys


Fingers crossed that those exceptions will work indefinitely. Thanks to scottgus1 for suggesting these exceptions - hopefully this will help others who are experiencing the same issue
Post Reply