[Fixed] Host crash with OS Dell PowerEdge 6850/EL7

[DaveWoodman]

Hi,

I have a Dell 6850 / 64GB / 4x Tulsa dual-core. OS is CentOS 7.1. RC2 + extensions installed. BIOS is most up to date available.

Whenever I attempt to start a VM the system crashes (hard). Vbox.log for VM is empty.

4.3.28 functions correctly.

Summary crash report below (trimmed since URLS are not permitted day 1):-

Code: Select all

time:           Fri 03 Jul 2015 13:37:22 BST
uid:            0 (root)
count:          1
Directory:      /var/spool/abrt/vmcore-127.0.0.1-2015.07.03-13:33:39
analyzer:       vmcore
architecture:   x86_64
component:      kernel
event_log:
kernel:         3.10.0-229.el7.x86_64
last_occurrence: 1435927042
os_release:     CentOS Linux release 7.1.1503 (Core)
runlevel:       N 3
type:           vmcore
username:       root
uuid:           8392523b65ce93455402683556fd750cd813fef5

vmcore:         Binary file, 334054418 bytes
vmcore-dmesg.txt: Text file, 87554 bytes

backtrace:
:
:Version: 3.10.0-229.el7.x86_64
:BUG: unable to handle kernel NULL pointer dereference at 0000000000000410
:IP: [<ffffffffa08bf7c5>] 0xffffffffa08bf7c4
:PGD f98bcd067 PUD f93353067 PMD 0
:Oops: 0002 [#1] SMP
:Modules linked in: fuse unix_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd sunrpc fscache ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security vboxdrv(OF) ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables iTCO_wdt iTCO_vendor_support dcdbas kvm_intel kvm lpc_ich mfd_core pcspkr serio_raw ipmi_si ipmi_msghandler shpchp uinput xfs libcrc32c sr_mod sd_mod cdrom ata_generic crc_t10dif pata_acpi crct10dif_common radeon
: i2c_algo_bit drm_kms_helper ttm drm ata_piix libata i2c_core tg3 e1000 ptp megaraid_sas pps_core floppy dm_mirror dm_region_hash dm_log dm_mod
:CPU: 6 PID: 7228 Comm: EMT-0 Tainted: GF          O--------------   3.10.0-229.el7.x86_64 #1
:Hardware name: Dell Computer Corporation PowerEdge 6850/0RD318, BIOS A06 09/02/2008
:task: ffff8810132f71c0 ti: ffff881018fb0000 task.ti: ffff881018fb0000
:RIP: 0010:[<ffffffffa08bf7c5>]  [<ffffffffa08bf7c5>] 0xffffffffa08bf7c4
:RSP: 0018:ffff881018fb3c80  EFLAGS: 00010246
:RAX: 0000000000000000 RBX: ffffc90016901000 RCX: 0000000000000000
:RDX: 0000000000000080 RSI: 0000000000000001 RDI: 0000000000004010
:RBP: ffff881018fb3d00 R08: 0000000000000000 R09: ffffc900169010f8
:R10: ffff881018fb3df0 R11: ffffffffa08b6440 R12: ffffc900168e1000
:R13: ffff881018fb3d40 R14: ffffc900168fc000 R15: ffffc900168e1000
:FS:  00007f7c1cc2a700(0000) GS:ffff88102fcc0000(0000) knlGS:0000000000000000
:CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
:CR2: 0000000000000410 CR3: 0000000f8e274000 CR4: 00000000000027e0
:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
:Stack:
: ffff881018fb3ca0 ffffffff8115659e 0000000000000189 ffff880f98ac3a00
: 0000000000001000 000000000000008b ffff881018fb3cd0 ffffffff811be516
: ffffea003df3dc80 00007f7c00000000 ffffea003df3dc80 ffffc900168fc000
:Call Trace:
: [<ffffffff8115659e>] ? __find_get_page+0x1e/0xa0
: [<ffffffff811be516>] ? mem_cgroup_update_page_stat+0x16/0x50
: [<ffffffffa04d6917>] ? supdrvIOCtlFast+0x77/0xa0 [vboxdrv]
: [<ffffffffa04d349b>] ? VBoxDrvLinuxIOCtl_5_0_0+0x5b/0x260 [vboxdrv]
: [<ffffffff811da135>] ? do_vfs_ioctl+0x2e5/0x4c0
: [<ffffffff811da3b1>] ? SyS_ioctl+0xa1/0xc0
: [<ffffffff81614a29>] ? system_call_fastpath+0x16/0x1b
:Code: 93 e8 01 00 00 45 31 c0 31 c9 be 80 00 00 c0 4c 89 f7 e8 ff 7b ff ff 85 c0 0f 88 ad f9 ff ff 49 8b 86 60 02 00 00 ba 80 00 00 00 <0f> ab 90 00 04 00 00 0f ab 90 00 0c 00 00 e9 a6 fd ff ff 0f 1f
:RIP  [<ffffffffa08bf7c5>] 0xffffffffa08bf7c4
: RSP <ffff881018fb3c80>

machineid:
:systemd=8da514bcd4c14485a1b903bb08e1bd1f
:sosreport_uploader-dmidecode=d0bd4487452614eb47afbc8469d58672a0fc102ba3d7dc65baa5675c84ac04de

Thanks,

Dave.

[DaveWoodman]

RC3 also exhibits the same behaviour - the host OS crashes and reboots whenever starting a VM (of any flavour).

VirtualBox 5 is therefore unusable on this platform as it stands.

The error report for RC3 follows (essentially the same error as with RC2).

I am happy to provide any further diagnostic information on request (but note that the vbox log is empty!)

Code: Select all

id faed546e22cdc2bc39cf71642d3d117d41dcf126
time:           Mon 06 Jul 2015 11:08:50 AM BST
uid:            0 (root)
count:          1
Directory:      /var/spool/abrt/vmcore-127.0.0.1-2015.07.06-11:05:05
analyzer:       vmcore
architecture:   x86_64
component:      kernel
event_log:
kernel:         3.10.0-229.el7.x86_64
last_occurrence: 1436177330
os_release:     CentOS Linux release 7.1.1503 (Core)
runlevel:       N 3
type:           vmcore
username:       root
uuid:           7cfabe4bbed9f1573c391e7ec7196adeb031fa32

vmcore:         Binary file, 316259245 bytes
vmcore-dmesg.txt: Text file, 87579 bytes

backtrace:
:
:Version: 3.10.0-229.el7.x86_64
:BUG: unable to handle kernel NULL pointer dereference at 0000000000000410
:IP: [<ffffffffa08eb765>] 0xffffffffa08eb764
:PGD f8d92b067 PUD f7b22f067 PMD 0
:Oops: 0002 [#1] SMP
:Modules linked in: vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) fuse unix_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd sunrpc fscache ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables iTCO_wdt iTCO_vendor_support dcdbas kvm_intel kvm lpc_ich mfd_core serio_raw pcspkr ipmi_si ipmi_msghandler shpchp uinput xfs libcrc32c sr_mod sd_mod cdrom ata_generic crc_t10dif pata_acpi crct10dif_common radeon
: i2c_algo_bit drm_kms_helper ttm drm ata_piix libata i2c_core tg3 e1000 megaraid_sas ptp pps_core floppy dm_mirror dm_region_hash dm_log dm_mod
:CPU: 6 PID: 30256 Comm: EMT-0 Tainted: GF          O--------------   3.10.0-229.el7.x86_64 #1
:Hardware name: Dell Computer Corporation PowerEdge 6850/0RD318, BIOS A06 09/02/2008
:task: ffff8810164f2220 ti: ffff881015da8000 task.ti: ffff881015da8000
:RIP: 0010:[<ffffffffa08eb765>]  [<ffffffffa08eb765>] 0xffffffffa08eb764
:RSP: 0018:ffff881015dabc80  EFLAGS: 00010246
:RAX: 0000000000000000 RBX: ffffc90016d65000 RCX: 0000000000000000
:RDX: 0000000000000080 RSI: 0000000000000001 RDI: 0000000000004010
:RBP: ffff881015dabd00 R08: 0000000000000000 R09: ffffc90016d650f8
:R10: ffff881015dabdf0 R11: ffffffffa08e23e0 R12: ffffc90016d45000
:R13: ffff881015dabd40 R14: ffffc90016d60000 R15: ffffc90016d45000
:FS:  00007f0ef3054700(0000) GS:ffff88102fcc0000(0000) knlGS:0000000000000000
:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
:CR2: 0000000000000410 CR3: 0000000f563bb000 CR4: 00000000000027e0
:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
:Stack:
: ffff881015dabca0 ffffffff8115659e 0000000000000189 ffff880f42093300
: 0000000000025260 000000000000008b ffff881015dabcd0 ffffffff811be516
: ffffea003de9be00 00007f0f00000000 ffffea003de9be00 ffffc90016d60000
:Call Trace:
: [<ffffffff8115659e>] ? __find_get_page+0x1e/0xa0
: [<ffffffff811be516>] ? mem_cgroup_update_page_stat+0x16/0x50
: [<ffffffffa075f917>] ? supdrvIOCtlFast+0x77/0xa0 [vboxdrv]
: [<ffffffffa075c49b>] ? VBoxDrvLinuxIOCtl_5_0_0+0x5b/0x260 [vboxdrv]
: [<ffffffff811da135>] ? do_vfs_ioctl+0x2e5/0x4c0
: [<ffffffff811da3b1>] ? SyS_ioctl+0xa1/0xc0
: [<ffffffff81614a29>] ? system_call_fastpath+0x16/0x1b
:Code: 93 e8 01 00 00 45 31 c0 31 c9 be 80 00 00 c0 4c 89 f7 e8 ff 7b ff ff 85 c0 0f 88 ad f9 ff ff 49 8b 86 60 02 00 00 ba 80 00 00 00 <0f> ab 90 00 04 00 00 0f ab 90 00 0c 00 00 e9 a6 fd ff ff 0f 1f
:RIP  [<ffffffffa08eb765>] 0xffffffffa08eb764
: RSP <ffff881015dabc80>

machineid:
:systemd=8da514bcd4c14485a1b903bb08e1bd1f
:sosreport_uploader-dmidecode=d0bd4487452614eb47afbc8469d58672a0fc102ba3d7dc65baa5675c84ac04de

os_info:
:NAME="CentOS Linux"
:VERSION="7 (Core)"
:ID="centos"
:ID_LIKE="rhel fedora"
:VERSION_ID="7"
:PRETTY_NAME="CentOS Linux 7 (Core)"
:ANSI_COLOR="0;31"
:CPE_NAME="cpe:/o:centos:centos:7"
:HOME_URL="https://www.centos.org/"
:BUG_REPORT_URL="https://bugs.centos.org/"
:
:CENTOS_MANTISBT_PROJECT="CentOS-7"
:CENTOS_MANTISBT_PROJECT_VERSION="7"
:REDHAT_SUPPORT_PRODUCT="centos"
:REDHAT_SUPPORT_PRODUCT_VERSION="7"
:

[frank]

You write that there is no VBox.log so I assume your host is crashing quite quickly when you try to start a VM. Does this also happen for "dummy" VMs, ie a VM which does not have any hard disk attached? And does it also happen if you disable VT-x in the VM settings?

[DaveWoodman]

Hi Frank - thanks for the reply

Very quickly, yes! Splash screen & rapid onset of death

I attempted to boot a 32-bit LiveCD (WanEM in this case) - no issues
As I live in a 64bit world, I built a 32bit Win 2008 for testing, and disabled VT-x - no issues
Enabling VT-x however, kills the host.
Booting a 64-bit LiveCD (systemrescue) also kills the host - of course VT-x is required here...

This processor does support VT-x, and version 4.3.23 worked without issues.

In case the 32-bit boot will tell you something, it is attached...

Dave.

[michaln]

The only thing that jumps out is the antiquated NetBurst CPU... whose virtualization performance is poor compared to the Core 2 generation and abysmal compared to newer offerings. Do you have any idea if the problem is specific to the hardware or the host OS? Did you happen to try the first 5.0 beta, on the off chance that it didn't have the same problem? (Then it would be a lot easier to pin it down.)

[michaln]

OK, we probably know what the problem is... no guarantees it's the only one though. We can't test it easily.

[DaveWoodman]

I did not try the BETA 1 at the time...

But, since it is still available on download site, I have tried it.

BEAT 1 behaviour is the same - the host goes away as expected.

There's still a lot of NetBurst out there, I suspect! I can understand if you don't have access to a relevant system, but i can offer access, by arrangement, to this one during UK working hours should you wish.

Alternatively I am happy to test whatever you might want to throw my way, and feedback.

[frank]

Could you check if this build fixes your problem? Thank you!

[michaln]

BEAT 1 behaviour is the same - the host goes away as expected.

Thanks for testing that! Now it seems it's a bug introduced at some point earlier in the 5.0 development cycle.

There's still a lot of NetBurst out there, I suspect!

No, there isn't, or at least not running VirtualBox... we've had these kinds of problems before, and it always took a while before they got spotted. Those old systems are really bad at hardware virtualization. In 2006, VT-x was basically a prototype, with extremely limited capabilities and poor (slow) implementation. Try running a post-2010 3GHz system and you won't touch NetBurst with a 10-foot pole. The difference isn't nearly as great in other areas but VT-x has really come a long way since those times. Nested paging together with TLB tagging greatly improves performance across the board, and Intel over time tuned the implementation so that even with the same features, VT-x is now several times faster than it used to be. Also the oldest systems with no support for APIC virtualization are absolutely terrible for SMP guests. Yours might not have that particular problem but we'd need a VBox.log with VT-x enabled to tell (Intel never published much data about what processors support which features exactly).

Anyway, thanks for discovering this problem early

[DaveWoodman]

Could you check if this build fixes your problem? Thank you!

It would appear so to do! Many thanks indeed!

...we'd need a VBox.log with VT-x enabled to tell ...

Your wish is my command - one is duly attached, now that I can provide it.

Thank you both very much - my home lab will be forever in your debt...

Was the change to disable paravirtualisation, by the way?

[michaln]

Thanks for the confirmation!

Looking at the new log file I can say that yes, this is one of those very old VT-x implementations I was talking about. Something like Windows XP with I/O APIC enabled will run terribly, even with a single virtual CPU. 64-bit guests are not so badly affected though.

And no, the change had nothing to do with paravirtualization... there was just a missing check in one place that was needed to deal with VT-x implementations without the "MSR bitmaps" feature. This passed unnoticed because we don't have any test system that's old enough. All the Core generation CPUs (2006 and later) have that VT-x feature, even if they lack many others.

Like I said, your CPU has essentially a prototype VT-x implementation. It's a typical Intel approach -- the implementation is functional and does what it promises, but it's very limited, very slow, and customers had to pay extra to get any VT-x at all.

[johnlee]

Is this fix now in the svn? Can't see it in the changesets that just came out today Wed pm - is it hidden is one with another name? John

[michaln]

https://www.virtualbox.org/changeset/56823/vbox