PFSense guest on Windows Host crashes

Discussions about using non Windows and Linux guests such as FreeBSD, DOS, OS/2, OpenBSD, etc.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

PFSense guest on Windows Host crashes

Post by User2358 »

Hi

I have a guest PFSense machine that crashes with the message

Code: Select all

---------------------------
VirtualBoxVM.exe - Application Error
---------------------------
The instruction at 0x00007FFDA106D556 referenced memory at 0x0000000000000000. The memory could not be read.


Click on OK to terminate the program
---------------------------
OK   
---------------------------
Nothing of note in logs, made absolutely fresh install of Windows Server 2019, still has the same crash.

The machine only seems to crash under load, like torrents for example.
When shutting down the host, the Virtualbox network interface is always busy, if that is relevant.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: PFSense guest on Windows Host crashes

Post by scottgus1 »

Start the guest from full power off, not save-state. Run until you see the problem happen, then shut down the guest from within the guest OS if possible. If not possible, close the Virtualbox window for the guest with the Power Off option set.

Please right-click the guest in the main Virtualbox window's guest list, choose Show Log. Save the far left tab's log, zip the log file, and post the zip file, using the forum's Upload Attachment tab.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

It's a hard crash for the guest, so, there's no shutting it down, just clicking the OK button, and the guest aborting.
I'll get it to crash shortly, just checking that this doesn't change anything.
Last edited by scottgus1 on 30. Dec 2020, 18:36, edited 1 time in total.
Reason: Took out quote of complete previous post - Please use Post Reply
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: PFSense guest on Windows Host crashes

Post by scottgus1 »

Any log might show the thing I'm thinking, due to the apparent Windows Server 2019 host OS.

Please right-click the guest in the main Virtualbox window's guest list, choose Show Log.

Search the far left tab's log for this text:

Attempting fall back to NEM

If you find it, Hyper-V is enabled and needs to be disabled. See HMR3Init: Attempting fall back to NEM (Hyper-V is active). Version 6.1.16 might fix this. If you're on an earlier version, try it.

If you don't find that text, save the far left tab's log, zip the log file, and post the zip file, using the forum's Upload Attachment tab.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

Hyper-V isn't even installed. I literally just reinstalled WinServ, installed Vbox, and made a new PFSense guest.
Otherwise, absolutely default install.

I'll hopefully have a log for you soon, but the ones I do have don't say anything about falling back to NEM.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

It finally crashed, so here's all the logs.

I couldn't close the guest via the power off option, as explained, the guest crashes entirely.
Last edited by User2358 on 4. Jan 2021, 10:13, edited 1 time in total.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: PFSense guest on Windows Host crashes

Post by scottgus1 »

You have a hardening error happening, with an exit code 0xc0000005. See Diagnosing VirtualBox Hardening Issues. Though that problem typically centers around graphics drivers, your vbox.logs have lots of errors over the audio drivers. Since routers typically don't play audio, try disabling audio completely in the VM's Audio settings. Also a fresh install of the router OS thereafter might be needed. Further try upgrading Virtualbox to 6.1.16 from the 6.1.12 you have now.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

I disabled audio, it'll probably resolve the issue as the Windows Audio Service doesn't run on that machine, thanks.

Although, the VirtualBox Interface process has ongoing connections when the guest crashes, and they never stop/close no matter what.
fth0
Volunteer
Posts: 5661
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: PFSense guest on Windows Host crashes

Post by fth0 »

User2358 wrote:Although, the VirtualBox Interface process has ongoing connections when the guest crashes, and they never stop/close no matter what.
I don't understand what you mean by that. Can you elaborate a bit?
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

fth0 wrote:I don't understand what you mean by that. Can you elaborate a bit?
If the guest container crashes, the Interfaces process will halt host shutdown by being too "busy", and this can be hours after crashing.
I forget where I got the information (where exactly in the machine), but I found out that it has "open connections". I'm not sure if they ever close, without the host restarting.
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: PFSense guest on Windows Host crashes

Post by scottgus1 »

If the VM continues to crash after disabling audio completely, please post a new hardening log and vbox.log showing the VM crash event.
fth0
Volunteer
Posts: 5661
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: PFSense guest on Windows Host crashes

Post by fth0 »

I know several VirtualBox processes and drivers, but I've never heard about a VirtualBox Interface process before. So I consulted the VirtualBox source code, just to learn that the VBoxSVC process is also called VirtualBox Interface. You never stop learning. ;)

Your primary problem is the crash inside the VirtualBoxVM process, which happens inside one of its core parts named Virtual Machine Monitor (VBoxVMM.dll). Your secondary problem is probably a connection between the died VirtualBoxVM process and the VBoxSVC process that is never cleaned up after the crash.

The crash information in the VBox.log file doesn't tell me enough about its cause, so I can only suggest to further investigate by trial and error. Use the current VirtualBox version 6.1.16 and eliminate all unnecessary VM components from the VM.
User2358
Posts: 8
Joined: 30. Dec 2020, 15:25

Re: PFSense guest on Windows Host crashes

Post by User2358 »

Just crashed again :<

Is there an experimental version of Vbox that offers more verbose logging?
Attachments
Router-2021-01-06-12-28-43.zip
(44.81 KiB) Downloaded 10 times
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: PFSense guest on Windows Host crashes

Post by scottgus1 »

This appears to have been a hard crash, 52-ish hours after start:
00:18:31.367755 GUI: UIMediumEnumerator: Medium-enumeration finished!
52:31:34.332090
52:31:34.332092 !!! rtR3WinUnhandledXcptFilter caught an exception on thread 0000000000000808!!!
52:31:34.332103
52:31:34.332103 ExceptionCode=0xc0000005 ExceptionFlags=0x00000000 ExceptionAddress=00007ffda03ad8b6
The exception code is the same as the hardening log exit code:
b10.744: supR3HardNtChildWaitFor[1]: Quitting: ExitCode=0xc0000005 (rcNtWait=0x0, rcNt1=0x0, rcNt2=0x103, rcNt3=0x103, 189189334 ms (=52.55 hours, fitting the run time on the crash), the end)
So you appear to still be having an intermittent hardening error on your host PC. The audio errors are gone, good. What's causing the crash is a bit beyond me, I can't see the cause in the logs, which sometimes happens: the cause doesn't get noticed by Virtualbox, only the crash. Better eyes than mine will have to help now, and I'll stay tuned to learn, too.
fth0
Volunteer
Posts: 5661
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: PFSense guest on Windows Host crashes

Post by fth0 »

I've analyzed the crash information from the latest VirtualBox 6.1.16 log files, but AFAICS, they unfortunately do not reveal the real source of the problem. Some bits and pieces in case a VirtualBox developer reads this:

The EMT thread crashed with the following call stack: vmR3EmulationThreadWithId() > EMR3ExecuteVM() > VMMR3HmRunGC() > vmmR3ServiceCallRing3Request() > PDMR3CritSectEnterEx() > PDMCritSectEnterDebug(). In VMMR3HmRunGC(), the previous call to SUPR3CallVMMR0Fast() prepared the call to PDMR3CritSectEnterEx(), but the Ring-3 address was 0 and finally led to the crash. From the VirtualBox source code, I can make the educated guess that the order came from pdmCritSectEnter(), where MMHyperCCToR3() was called to obtain the Ring-3 address from a Ring-0 address, and it has a developer's prophecy: "You'll be damned if this is not in the HMA! :-)". Gotcha! Since the Ring-0 address was not revealed, I cannot guess the physical or virtual hardware component involved (with the exception that it's handled by the PDM), so this seems to be a dead end at the moment.
User2358 wrote:Is there an experimental version of Vbox that offers more verbose logging?
The release version of VirtualBox and (above all) a self-built debug version of VirtualBox both offer more verbose logging. Note that it requires detailed knowledge about the inner workings of the VirtualBox code and is therefore rather suited for developers IMHO. For details, see Technical documentation and The VirtualBox/IPRT logging facility.
scottgus1 wrote:The exception code is the same as the hardening log exit code: [...]0xc0000005
In case you don't know it already: 0xc0000005 (STATUS_ACCESS_VIOLATION) is the Windows error code for one of the most common programming mistakes in the C and C++ programming languages, sometimes called Null pointer access. VirtualBox captures the Windows exception triggered in its own code and writes corresponding information to the VBox.log and VBoxHardening.log files. It is quite possible that the crash has no hardening background in the typical sense (no 3rd-party software involved).
Post Reply