Solaris 10 host panic, vboxdrv:supdrvIOCtlFast

[Forest4Trees]

Our host O/S experienced a panic, and Oracle Support identified VirtualBox as the cause, since it had vboxdrv:supdrvIOCtlFast and vboxdrv:VBoxDrvSolarisIOCtl in the stack trace.

We did get a kernel dump as well, but it is huge (20GB).

Can anyone help us? Is this a new bug?

Here is our host configuration:

• Hardware: Oracle X4270 M2 (x86)
• CPU: 2 x Intel Xeon X5690, 3.47GHz
• RAM: 245GB
• Host O/S: Solaris 10 (x86_64), Update 11, Kernel 147148-26
• VirtualBox: 4.3.2
• Guest O/S: Windows Server 2012 (VM was running, under normal load during the panic)

Excerpt from /var/adm/messages:

Code: Select all

Dec  2 10:09:01 server4 unix: [ID 836849 kern.notice]
Dec  2 10:09:01 server4 ^Mpanic[cpu15]/thread=fffffe8001004c40:
Dec  2 10:09:01 server4 genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=fffffe8010c699f0 addr=28 occurred in module "<unknown>" due to a NULL pointer dereference
Dec  2 10:09:01 server4 unix: [ID 100000 kern.notice]
Dec  2 10:09:01 server4 unix: [ID 839527 kern.notice] sched:
Dec  2 10:09:01 server4 unix: [ID 753105 kern.notice] #pf Page fault
Dec  2 10:09:01 server4 unix: [ID 532287 kern.notice] Bad kernel fault at addr=0x28
Dec  2 10:09:01 server4 unix: [ID 243837 kern.notice] pid=0, pc=0xfffffffff83559db, sp=0xfffffe8010c69ae0, eflags=0x10282
Dec  2 10:09:01 server4 unix: [ID 211416 kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 26f8<xmme,fxsr,pge,mce,pae,pse,de>
Dec  2 10:09:01 server4 unix: [ID 354241 kern.notice] cr2: 28 cr3: 2ffd6ec000 cr8: f
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   rdi: fffffe8001004c40 rsi:                4 rdx: fffffeda1ad5c030
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   rcx:                4  r8: fffffeda1ad35ae0  r9:   1920ba9f66006b
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   rax:                0 rbx: fffffeda1ad35ae0 rbp: fffffe8010c69af0
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   r10:          213cc45 r11:           989680 r12: fffffeda1ad2d000
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   r13:    21b06a5a9e527 r14:                1 r15:                0
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   fsb: fffffd7ffefc6200 gsb: fffffebe7219b800  ds:                0
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]    es:                0  fs:                0  gs:                0
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]   trp:                e err:                0 rip: fffffffff83559db
Dec  2 10:09:01 server4 unix: [ID 592667 kern.notice]    cs:               28 rfl:            10282 rsp: fffffe8010c69ae0
Dec  2 10:09:01 server4 unix: [ID 266532 kern.notice]    ss:               30
Dec  2 10:09:01 server4 unix: [ID 100000 kern.notice]
Dec  2 10:09:01 server4 genunix: [ID 655072 kern.notice] fffffe8010c69900 unix:die+da ()
Dec  2 10:09:01 server4 genunix: [ID 655072 kern.notice] fffffe8010c699e0 unix:trap+5e6 ()
Dec  2 10:09:01 server4 genunix: [ID 655072 kern.notice] fffffe8010c699f0 unix:cmntrap+140 ()
Dec  2 10:09:01 server4 genunix: [ID 802836 kern.notice] fffffe8010c69af0 fffffffff83559db ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69b40 fffffffff8383f14 ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69b60 fffffffff83839a8 ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69b80 fffffffff8302de0 ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69bb0 fffffffff82f3711 ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69c80 fffffffff82f328b ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69ca0 fffffffff8385767 ()
Dec  2 10:09:02 server4 genunix: [ID 802836 kern.notice] fffffe8010c69d00 fffffffff82fe735 ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69d10 vboxdrv:supdrvIOCtlFast+8c ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69db0 vboxdrv:VBoxDrvSolarisIOCtl+109 ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69dc0 genunix:cdev_ioctl+1d ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69de0 specfs:spec_ioctl+50 ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69e20 genunix:fop_ioctl+2e ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69f00 genunix:ioctl+ac ()
Dec  2 10:09:02 server4 genunix: [ID 655072 kern.notice] fffffe8010c69f10 unix:brand_sys_syscall+21d ()
Dec  2 10:09:02 server4 unix: [ID 100000 kern.notice]
Dec  2 10:09:02 server4 genunix: [ID 672855 kern.notice] syncing file systems...
Dec  2 10:09:02 server4 unix: [ID 836849 kern.notice]
Dec  2 10:09:02 server4 ^Mpanic[cpu15]/thread=fffffe8000fdac40:
Dec  2 10:09:02 server4 genunix: [ID 683410 kern.notice] BAD TRAP: type=8 (#df Double fault) rp=fffffebe7225bf10 addr=0
Dec  2 10:09:02 server4 unix: [ID 100000 kern.notice]
Dec  2 10:09:02 server4 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
Dec  2 10:09:01 server4 genunix: [ID 100000 kern.notice]
Dec  2 10:09:01 server4 genunix: [ID 665016 kern.notice] ^M100% done: 18050569 pages dumped,
Dec  2 10:09:01 server4 genunix: [ID 851671 kern.notice] dump succeeded

[Ramshankar]

Could you attach the matching VBox.log for the VM when this panic happened?

Also, if this is reproducible, does 4.3.4 fix the problem?

[Forest4Trees]

Correction: BTW, I made a mistake in the original post. The guest VM is actually running Windows Server 2008 R2, SP1, not Windows Server 2012.

Recreating The Problem: We have been running this guest VM under 4.3.2 for 4 weeks now, and this is the first incident like this. Only time will tell, whether this is repeatable. It may take months, or never.

Upgrading: Since 4.3.4 just came out, I was hoping to delay upgrading for a few weeks, to see if others encounter any issues with the new version. If we do upgrade to 4.3.4, we may not be able to determine whether or not it is repeatable on 4.3.2. However, I'm open to suggestions.

Log: I found the VBox.log, and after sanitizing sensitive information, and cutting out 2000 lines of "LsiLogic" lines, I've uploaded the file. I do see a few "pointer <hex address>/<number> is out of range" messages.

VBox.log

VirtualBox guest VM log.

(69.46 KiB) Downloaded 35 times

[Ramshankar]

Upgrading: Since 4.3.4 just came out, I was hoping to delay upgrading for a few weeks, to see if others encounter any issues with the new version. If we do upgrade to 4.3.4, we may not be able to determine whether or not it is repeatable on 4.3.2. However, I'm open to suggestions.

No, please try 4.3.4. We fixed a bug in 4.3.4 related to preemption that can cause host panics/hangs (on Linux and Solaris hosts) and chances of that bug hitting seems more slim on Solaris. In the mean time I'll try take a look at the existing panic information you have provided here. Thanks for the report.

[Forest4Trees]

We upgraded to VirtualBox 4.3.4, and have been running that for a week. So far, there have been no host O/S panics.