VM freezes when activating 64-bit mode

[Neolander]

Hello everyone !

I guess you don't see many hobby OS developers around here... anyway, here I go.

I'm developing a 64-bit kernel. It runs fine on Bochs and Qemu no matter what the underlying OS is, and it used to run fine on VirtualBox 3.0/Windows 7 x64 with VT-x enabled. My computer is an Asus N61JV laptop with a Core i5 430M and 4GB of DDR3.

Now, on the other hand, neither VirtualBox 4.0 on Windows 7 x64 (I upgraded the previous install) nor VirtualBox 3.2.10 OSE on Fedora 14 x64 will accept to boot it at all. Both just freeze during long mode activation, and then keep using 100% CPU without doing anything else. The thing is, I don't know why.

During said boot process, I noticeably check through CPUID that
-PAE and APIC are available
-64-bit mode and NX/DEP are supported
Both checks are passed.

Then I setup page tables and things like that, and when I'm ready to go in the 32-bit "compatibility" subset of long mode, I do so using the following code (in AT&T syntax, sorry)

Code: Select all

  /* At this point, we know that long-mode support is available
     Step 1 : Enable PAE */
  mov   %cr4, %eax
  bts   $5,   %eax
  mov   %eax, %cr4

  /* Step 2 : Load CR3 value */
  mov   4(%ebp), %eax
  mov   %eax, %cr3

  /* Step 3 : Set LME and NXE bits in the EFER Model Specific Register */
  mov   $0xc0000080, %ecx
  rdmsr
  bts   $8,   %eax
  bts   $11,   %eax
  wrmsr

/* Enable paging, then effectively enable long mode (we'll be in 32-bit compatibility mode at this point)  */
  mov   %cr0, %eax
  bts   $31,  %eax
  mov   %eax, %cr0
  ljmp $24, $compatibility_mode
compatibility_mode:
  mov   $0, %eax

The VM stops working while enabling paging, before the long jump.
Here are the logs : http://pastebin.com/nYyeB6Gv

Any idea ?