I think I'm having the same problem as you, or at least a very similar one, and have spent much of the weekend trying to isolate it as well. I had a 64-bit Windows Server 2008 R2 virtual machine that ran fine in VirtualBox 3.2.x, but after upgrading to 4.0 at the beginning of the year I consistently got blue screens when performing certain actions. I am now working with a completely new virtual machine created in VirtualBox 4.0.8 and that uses all of the defaults of the Windows2008_64 OS type and has a plain vanilla installation of 64-bit Windows Server 2008 R2 with Service Pack 1 (no additional applications or Windows components installed)...but am
still having the exact same problems.
What happens is that when I do any of the following...
- Shut down or reboot the guest
- Connect to Windows' (not VirtualBox's) Remote Desktop server, then log out from the remote session
- Run Process Explorer, then exit it
...I get a blue screen.
The first one would happen with the old virtual machine every single time I tried to shut down or restart Windows. I'd go to shut down Windows and it would appear to go through the entire shut down sequence, but then it'd be back at the BIOS screen and boot back into Windows. The next time I'd log in, there'd be a message that a blue screen had occurred, even though I could never actually see the blue screen (even after disabling System Properties => Startup and Recovery => System failure => Automatically restart). The only way I could get the guest to power off was to initiate a shut down, let it blue screen and reset, then power off the guest at the BIOS screen. I do not remember what the stop code was (I do recall seeing IRQL_NOT_LESS_OR_EQUAL at some point but I can't say for sure), and I have not yet observed this behavior with the new virtual machine.
Logging out of a Remote Desktop session would cause the guest to reset or lock up every single time on the old virtual machine, and probably once every two or three times on the new one. I can't say what effect Process Explorer had on the old virtual machine, but on the new one if I just run Process Explorer with its default configuration and exit it without even doing anything then, again, once every two or three times it will cause a blue screen. With both logging out of Remote Desktop and exiting Process Explorer on the new virtual machine the stop code for the blue screen is 0x0000001a (MEMORY_MANAGEMENT).
I thought maybe I'd narrowed this apparent bug down to version 4.0.x of VirtualBox Guest Additions because I wasn't having problems after installing 3.2.12, but after playing with it some more I do get the same blue screens under the same conditions with 3.2.12. Then I thought maybe it was the Guest Additions in general, because after reverting back to a snapshot taken just after Windows installation and with no Guest Additions installed things seemed to be running better...but no, I've been able to reproduce the same blue screen with no Guest Additions at all.
I ran Memtest86+ overnight recently and it came back with no errors, so I don't think it's a hardware problem. I also had, at one time, a couple 64-bit Ubuntu Server guests running alongside the old Windows virtual machine and those didn't experience any issues like this. I'm tempted to revert back to VirtualBox 3.2 just to see if this problem goes away, but I've already had the 3.x to 4.x virtual machine directory format conversion trash three virtual machines of mine, and I'm not keen on going through that again in the event I can create a working, non-bluescreening 3.x Windows Server 2008 virtual machine. This does seem like a bug in 4.0, though I also tried analyzing one of the minidumps that were created and it, too, pointed to ntoskrnl.exe, so I don't know if that rules out VirtualBox as the culprit or not.