virtualbox.org

VB aside for a moment, on a fairly regular basis, I used to get things like the following in kern.log: One by one the CPUs would lock up, until the whole computer was frozen such that I couldn't do anything, neither from the console nor remotely (ssh), and just had to manually poweroff and reboot.

I made sure BIOS and kernel were up to date, etc., etc. No resolution. It did seem that the problem happened mostly, if not exclusively, shortly after or while running my VirtualBox Windows VM. I changed a couple BIOS settings and now have gone a long time with no kernel lockup, whether I used the Windows VM extensively or not. The most notable BIOS setting I changed was to disable the Execute Disable bit.

I'm not a hardware or deep virtualization guy, so I'm reaching above my head here, but my "naive" theory goes like this: The Windows guest is setting some region of memory to no-execute, but the Linux host somehow "doesn't get the message". Later, either because the VM shutdown or something else, the kernel wants to use that memory for processing but runs into the unexpected NX bit set that was set by the guest.

Like I said, I know I don't know what I'm talking about here, but it seems a plausible theory based on my limited knowledge and the observed patterns. Somehow, with NX bit set, kernel lockup would follow in a relatively short time seeming to follow running the Windows VM.

Specs:
Guest: Windows 7 Professional, SP1, 64-bit, 5016MB Ram allocated to VM
VB: 5.0.24 (Debian package 5.0.24~dfsg-1~bpo8+1), Extension pack and guest additions installed, Enable PAE/NX is checked(!!!), Enable VT-x and Nested paged are checked
Host: Debian 8 (jessie), "Linux 4.5.0-0.bpo.2-amd64 #1 SMP Debian 4.5.4-1~bpo8+1 (2016-05-13) x86_64 GNU/Linux"
Hardware: Intel(R) Core(TM) i7 CPU; Toshiba Satellite A665; 8GB ram
BIOS: VT-x is enabled; Execute Disable (NX) bit alternately enabled then disabled (as described above); Bios version 2.3 for above noted laptop model

I have to admit I cannot explain your particular problem but if everything
is working as it should, the NX bit will not cause any problems.

Lets take simple example which hopefully I can explain well enough.

We have one Windows guest which sets the NX bit in the page table entry.
At this point the 4K area of real storage (as viewed by the guest) cannot
be used to execute instructions by the guest OS. This is the same as if the
OS was running native (ie not virtualised).
At the Host level this 4K area of storage will never be used to execute host
instuctions as it's allocated to the guest and forms part of the guests working set.
I've ignored the complexity of the two levels of page tables involved when
running VM's as that is difficult to explain and I'm trying to keep the explanation simple.

You then decide to destroy the guest. In the end, the Host will reset the
page table entry that referred to the guests 4K area of storage. At this point
that frame is available to be allocated again.

Note : In case you're unclear, a page refers to a 4K area of virtual storage,
a frame refers to a 4K area of real storage (RAM). It's the use of page tables
by a virtual storage OS (hypervisor or not) that allows the whole concept of
address spaces and by extension the ability of programs to be re-locatable
whilst the program itself views storage as a flat, contiguous area of storage.