[Solved] Memory Access error running on Windows Server 2012 R2

Discussions related to using VirtualBox on Windows hosts.
Post Reply
afoote3
Posts: 4
Joined: 11. Oct 2017, 14:49

[Solved] Memory Access error running on Windows Server 2012 R2

Post by afoote3 »

Hi all,

We’re having some odd problems running our testing server. I’ll do my best to explain the setup and the problem:

We have an automated testing server set up with Windows Server 2012 R2 host OS. On this server, we have an instance of Oracle VirtualBox installed (version 5.1.2.8956). Finally, on the tester’s computer, we are running HP Quality Center (version 9.2.0 – this is a very old version, but we are in the process of upgrading). All users are running Windows 7.

The tester starts up an automated test in Quality Center, which starts a PowerShell script on the server. This PowerShell script starts a virtual machine, using a VirtualBox COM object, by cloning an existing machine from a list of approved machines. Once the machine is cloned and started up, the specified test will then run on that virtual machine. Results are copied from the guest OS to the host OS, and the VM is removed.

The server handles multiple automated test requests at once. Each test request will run against a different VM. These tests are also not resource-bound (we have dual 6-core Xeon processors with 128GB RAM).

We are seeing issues that pop up with some regularity. These errors appear in the Windows Event Viewer as “Application Errors”, with Event ID 1000 and Task Category (100). An example event log looks like this:

Code: Select all

Faulting application name: VirtualBox.exe, version: 5.1.2.8956, time stamp: 0x5790f053
Faulting module name: VBoxC.dll, version: 5.1.2.8956, time stamp: 0x5790f057
Exception code: 0xc0000005
Fault offset: 0x00000000000a6a07
Faulting process id: 0xe34
Faulting application start time: 0x01d3410a03f156c8
Faulting application path: E:\VirtualBox\VirtualBox.exe
Faulting module path: E:\VirtualBox\VBoxC.dll
Report Id: 9a36be58-ad04-11e7-80f7-a0369f65611f
Faulting package full name: 
Faulting package-relative application ID:
According to the debug log, this exception code means that “the thread tried to read from or write to a virtual address for which it does not have the appropriate access.” This seems very odd because VirtualBox is being run as an administrator for each automated test.

However, there is another behavior that we have observed. It could be related, or it could be a red herring, and I haven’t figured out which yet. Anyway, The VirtualBox manager running on the server seems to slowly accumulate “inaccessible” virtual machines. For now, we simply clear these out periodically, but it doesn’t seem like this should happen at all.

Any advice for what to do here would be greatly appreciated.
Last edited by afoote3 on 27. Oct 2017, 15:27, edited 1 time in total.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Memory Access error running on Windows Server 2012 R2

Post by socratis »

A couple of notes. There is no 5.1.2 (8956) version, there is 5.1.2 (108956), it's funny that a "10" is chopped from the build field, even in the logs. That's really weird. Now, 5.1.2 is quite old, it came out on 2016-07-21, which is eons in software terms. Right now the current version is 5.1.28, I would suggest that you update to that one and see if the problem persists.

I've only found a somewhat similar problem to what you're describing in ticket #8298, but they don't mention at which version that specific problem was addressed. But given that the last comment is almost two years ago, one could safely assume that 5.1.2 should contain the fix, although I'm not willing to make a bet.

In any event, please update to the latest 5.1.x version, or even 5.2.0rc1 and see if the problem persists or not.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
afoote3
Posts: 4
Joined: 11. Oct 2017, 14:49

Re: Memory Access error running on Windows Server 2012 R2

Post by afoote3 »

Thanks for your suggestion! I looked through the change logs in the user manual for everything that's been updated since 5.1.2. It looks like there were a few fixes introduced that could have an impact here. I'll upgrade to 5.1.28 and see how that fares. Once I have solid evidence, I will post an update to say if this is the solution we are looking for.
afoote3
Posts: 4
Joined: 11. Oct 2017, 14:49

Re: Memory Access error running on Windows Server 2012 R2

Post by afoote3 »

The upgrade to 5.1.28 was not without issues. We have a PowerShell script that is supposed to power down a running VM, then unregister it from VirtualBox Manager, rename the storage, and move it to an archive. However, that process was not happening after the upgrade.

The power down function was successfully getting the machine into a powered down, unlocked state:

Code: Select all

        $virtualBox = New-Object -ComObject "VirtualBox.VirtualBox"
        $virtualBoxSession = New-Object -ComObject "VirtualBox.Session"
        $vm = $virtualBox.FindMachine($name)
        $result = AddMessageToLog "Powering down $name" $true
        $vm.LockMachine($virtualBoxSession,1)
        $shutdownProgress = $virtualBoxSession.Console.PowerDown()
        $shutdownProgress.WaitForCompletion(-1)
        $virtualBoxSession.UnlockMachine()
        $result = AddMessageToLog "Virtual machine powered down successfully" $true
But, after that, the machine was not being unregistered and moved. This was because there was still a running VirtualBox.exe process that was using or otherwise in contact with the VM storage files. To fix that, we had to use a PowerShell call to completely stop the running VirtualBox process, like so:

Code: Select all

    $processID = $vm.SessionPID
    AddMessageToLog "Ending VirtualBox process: $processID" $true
    $result = Stop-Process -id $processID -Force
    $vmMediaList = $vm.Unregister(4)
    $result = AddMessageToLog "Unregistering machine with VirtualBox Manager" $true
Now we have the system behaving as expected again. I wanted to include this here so that if someone in the future has the same unregistering issue we had, there is an answer for that problem.

More to the point of the thread, though, we have not yet seen another VirtualBox application error in the Windows Event Viewer since the upgrade was completed last Friday. It's only been a few days, but so far everything is looking good.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Memory Access error running on Windows Server 2012 R2

Post by socratis »

Great, thanks for the feedback. Not sure if I could mark this as [Solved] or not. If you feel comfortable, edit the title and add a [Solved] at the beginning, plus a note/post to let us know.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
afoote3
Posts: 4
Joined: 11. Oct 2017, 14:49

Re: Memory Access error running on Windows Server 2012 R2

Post by afoote3 »

I believe this issue can be safely marked as [Solved] at this point. We have not seen the issue described in the two weeks that have passed since the upgrade. Thanks again for your help, and I'm glad this was an easy one for you!
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: [Solved] Memory Access error running on Windows Server 2012 R2

Post by socratis »

No problem, and thanks once more for the feedback...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Post Reply