Repeatable Virtualbox VM Crash

[Chisleu]

Hello,

I use VirtualBox as a testing environment. I use Vagrant to create customized CentOS 7 VMs for phpunit testing.

We have always had an issue with the VM unexpectedly rebooting while testing. It was rare when I had the execution cap turned down to 50%, but it happened occasionally.

We turned up the execution cap, and threaded the phpunit tests so they would use both cores, but now the crashes happen much more often. Often times I can't get our few minutes of testing done without a crash.

It does not appear to be a kernel panic. klog is enabled and should be dumping something, but /var/crash is empty on boot.

When this happens, nothing seems to change in the Virtualbox interface. It appears to stay in the "Running" mode. It acts as if the VM reboots without warning. Vagrant is unaware of the system restart, so it does not conduct post-boot initialization (triggers plugin).

It is repeatable by running our parallel phpunit testing at most 2-3 times. It heats up the CPUs with ~75% CPU load (User / System).

It is affecting every development machine we have, macbook air, pro, Yosemite and El Capitan.

I'm at your disposal if there is any log file or other information I can provide to help diagnose the issue.

Thank you,
-Nick

[loukingjr]

Vagrant is not supported in this forum.

[Chisleu]

I understand that, but the issue I'm having is with a VM in Virtualbox rebooting. I installed CentOS in Virtualbox. The VM boots in Virtualbox, and it appears to crash/reboot without warning in Virtualbox.

I'm happy to confirm the issue exists without Vagrant involved if you would like. I can manually execute the few commands that Vagrant handles for me after booting the Virtualbox-controlled VM. Then run the testing until it crashes.

I would love to know if there is some information I can send when that happens. Is there a method to extract a log from Virtualbox?

[loukingjr]

Attach the vbox.log for the problematic guest.

With the VM fully shut down, right click it in the VB GUI. Select "Show Log" and save "VBox.log" (ONLY) to a zip file. Attach the zip here.

[Chisleu]

Very good! I'll do as I said and then attach the log.

[Chisleu]

It appears that Virtualbox is resetting the VM because it doesn't respond to some sort of heartbeat within a certain timeframe. This only happens when I'm cranking the CPU w/ high system usage in a parallel testing environment.

Here is the last frame of "top" before I lost contact with the system:

Code: Select all

top - 14:15:14 up 36 min,  2 users,  load average: 2.77, 2.08, 1.50
Tasks: 128 total,   2 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s): 41.8 us, 21.0 sy,  0.0 ni, 35.6 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
KiB Mem :  2916892 total,  1323604 free,   913820 used,   679468 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1797808 avail Mem

Here is the reason I think this might be the case:

Code: Select all

00:36:32.322976 PIT: mode=2 count=0x4a9 (1193) - 1000.15 Hz (ch=0)
00:36:32.483470 RTC: period=0x20 (32) 1024 Hz
00:36:34.220585 VMMDev: HeartBeatCheckTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago
00:36:42.749982 ACPI: Reset initiated by ACPI
00:36:42.750043 Changing the VM state from 'RUNNING' to 'RESETTING'
00:36:42.752995 APIC: Re-activating Local APIC
00:36:42.753018 CPUM: SetGuestCpuIdFeature: Enabled APIC
00:36:42.753028 PIT: mode=3 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:36:42.792893 AHCI#0: Reset the HBA
00:36:42.793000 PDMR3Reset: after    40 ms, 1 loops: 1 async tasks - piix3ide/0
00:36:42.793041 PIIX3 ATA: Ctl#0: finished processing RESET
00:36:42.793084 PIIX3 ATA: Ctl#1: finished processing RESET
00:36:42.827276 Changing the VM state from 'RESETTING' to 'RUNNING'
00:36:42.829235 VMMDev: Guest Log: BIOS: VirtualBox 5.0.6

I'm also attaching the full vmware log (zipped).

I have redacted all sensitive elements I could see, but please let me know if I missed something.

I really appreciate you taking the time to help diagnose this issue. I've spent many hours trying to discover the cause or narrow down a configuration cause of it. I did manage to get a kernel panic text in the console log earlier, but I was still unable to get a core dump to save to /var/crash. I made several changes in attempts to get CentOS7 to actually save the file, but I have been unsuccessful as of yet.

I expected to be able to replicate the kernel panic text output, but I've been unable to so far. It said it had to do with I/O ACHP Timing. This makes sense because it only started happening when we added multiple cores in initial attempts to speed up our unit testing through parallel processing.

[loukingjr]

there are a few problematic issues with the guest.

00:00:00.032694 Host RAM: 8192MB total, 4133MB available
00:00:00.147452 RamSize <integer> = 0x00000000c0000000 (3 221 225 472, 3 GB

The memory assigned to the guest is too great. Normally it should be no more than 50% of available memory. So 2GB would be better.
The CPU…

00:00:00.230725 Full Name: "Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz"

Has 2 cores and 4 threads. You have both cores assigned to the guest. It should be one core.

00:00:00.147774 VRamSize <integer> = 0x0000000000c00000 (12 582 912, 12 MB)

The VRAM should be set a little higher. Somewhere between 32MB and 64MB.

You have no extension pack installed so there won't be any USB support.

[Chisleu]

there are a few problematic issues with the guest.

00:00:00.032694 Host RAM: 8192MB total, 4133MB available
00:00:00.147452 RamSize <integer> = 0x00000000c0000000 (3 221 225 472, 3 GB

The memory assigned to the guest is too great. Normally it should be no more than 50% of available memory. So 2GB would be better.
The CPU…

00:00:00.230725 Full Name: "Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz"

Has 2 cores and 4 threads. You have both cores assigned to the guest. It should be one core.

00:00:00.147774 VRamSize <integer> = 0x0000000000c00000 (12 582 912, 12 MB)

The VRAM should be set a little higher. Somewhere between 32MB and 64MB.

You have no extension pack installed so there won't be any USB support.

Yes, I had it turned down to 2GB, but I turned it up when testing and didn't turn it back down for this log.

I have it set to 2 of 4 cores in the Virtualbox interface. I understand this is threads. I only see 2 threads when I 'cat /proc/cpuinfo'. My primary system stays responsive when fully loading the VM, so I'm not sure what is going on there. Is it really using all of my hyperthreads in the VM?

I will increase the VRAM. I left it where it is because It has no visual interface. I only SSH to it. I generally don't even show the GUI because it is easier to SSH and tunnel.

Thanks for the feedback!
-Nick

[loukingjr]

VirtualBox only cares about cores.

[Chisleu]

VirtualBox only cares about cores.

I'm interested why it shows 4 cores as the max (before it shows red on the slider.)

I have changed the logic to use 1/4 of the threads so it only uses 1 of my cores and 2 on the macbook pros.
I increased the VRAM to 32MB and reduced the RAM to 2GB.

I'll check back with success or fail! I really appreciate the help with this.

[loukingjr]

I'm interested why it shows 4 cores as the max (before it shows red on the slider.)

I've been asking the same question on and off for 6 years. I don't think I've received a clear answer.
BTW, since you aren't using a GUI 12MB is probably enough VRAM. If you only use one core I suspect you should be able to run the execution cap at 100%. All my guests do. Keep an eye on it though.

[Chisleu]

Interesting. It actually boots with 4 cores enabled and shows 4 cores in the Linux VM. It works until it crashes, but it only crashes under high load. With 2-3 cores enabled, the primary system still maintains responsiveness with 100% on 3 "cores". I guess I made assumptions based on what it says it was doing. I've not been able to crash it since, however it never crashed on 1 core. I will need to test it on 2 cores tomorrow on a macbook pro 4/8 core/thread system.

[Chisleu]

Same problem now confirmed on 4 core machine.

I got my Macbook Pro test machine today and it has a a i7-4980HQ, which is a 4 core 8 thread CPU.

We experienced the same kernel panic related to the APIC subsystem which crashes the VM as I was seeing before when I ran it on 2 cores myself.

I confirmed the MBPro had no other Virtualbox VMs, docker micro instance VM, or anything else of that nature running. It is on OSX Yosemite. My Air is on El Capitan.

I'm including the log file from that system's failure. It appears to do the same heartbeat failure that I experienced. The console scrolled the kernel panic information, but I did not take the extra step yet to log the console to serial port in hopes of getting the entire kernel panic console output. That is what I'm doing now.

I really appreciate all the help you people have given so far. Again, I'm happy to provide any assistance I can in helping to diagnose this issue.


EDIT:
For completeness, I'm including the console log captured just now when using 2 cores on my air. I will test with the Pro ASAP to confirm it is the same issue there. NOTE: The above log file is from the 4 core machine trying to run a 2 core VM.

Code: Select all

[ 1902.833377] general protection fault: 0000 [#1] SMP
[ 1902.847453] Modules linked in: bridge stp llc vboxsf(OF) binfmt_misc crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr serio_raw i2c_piix4 parport_pc video parport xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_common ata_generic pata_acpi ata_piix ahci libahci vboxvideo(OF) virtio_net drm virtio_pci virtio_ring vboxguest(OF) virtio i2c_core libata [last unloaded: ip_tables]
[ 1902.920541] CPU: 1 PID: 25436 Comm: php Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
[ 1902.935913] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 1902.945216] task: ffff880078d038e0 ti: ffff880000104000 task.ti: ffff880000104000
[ 1902.949390] RIP: 0010:[<ffffffffa006de96>]  [<ffffffffa006de96>] VbglR0HGCMInternalCall+0x426/0xe70 [vboxguest]
[ 1902.967956] RSP: 0018:ffff880000107b90  EFLAGS: 00010286
[ 1902.971296] RAX: 0000000000000000 RBX: ffff88007b157410 RCX: 0000000000bdc0ea
[ 1902.977277] RDX: 0000000000bdc0e9 RSI: 0000000000000000 RDI: 0000000000000000
[ 1902.994868] RBP: e287be0000000000 R08: 0000000000016420 R09: ffff88007fd16420
[ 1903.000360] R10: ffffea0000d9f7c0 R11: ffffffffa00719fa R12: 000007f800000040
[ 1903.019465] R13: 0000000400000000 R14: 0000439b00000000 R15: 0000500000000000
[ 1903.031343] FS:  00007fc8ffc68840(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[ 1903.044469] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1903.054685] CR2: 00007f0056258000 CR3: 000000001649b000 CR4: 00000000000406e0
[ 1903.056589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1903.063234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1903.067486] Stack:
[ 1903.071776]  01e10c00140f4375 01e10c001408cf16 01e10c001408cf16 008081a41408cf16
[ 1903.080787]  000001f500000002 0000000100000014 02329a2e01000004 0000000000000000
[ 1903.088719]  0000000000000000 00000000fffffffa ffff880000107c68 ffffffffa006ba62
[ 1903.093585] Call Trace:
[ 1903.094593]  [<ffffffffa006ba62>] ? VbgdCommonIoCtl+0x262/0x1c10 [vboxguest]
[ 1903.096575]  [<ffffffffa0068272>] ? VBoxGuestIDCCall+0x52/0x70 [vboxguest]
[ 1903.104967]  [<ffffffffa0289a76>] ? vbglDriverIOCtl+0x16/0x20 [vboxsf]
[ 1903.111258]  [<ffffffffa028a731>] ? VbglHGCMCall+0x41/0x90 [vboxsf]
[ 1903.121974]  [<ffffffffa028adee>] ? vboxCallCreate+0x8e/0xc0 [vboxsf]
[ 1903.126092]  [<ffffffff811e6814>] ? mntput+0x24/0x40
[ 1903.132028]  [<ffffffffa0288bcc>] ? sf_stat+0x6c/0x120 [vboxsf]
[ 1903.137330]  [<ffffffffa0288d14>] ? sf_inode_revalidate+0x94/0xf0 [vboxsf]
[ 1903.145731]  [<ffffffff811d7e82>] ? user_path_at_empty+0x72/0xc0
[ 1903.150635]  [<ffffffffa0288dbe>] ? sf_getattr+0x1e/0x40 [vboxsf]
[ 1903.155870]  [<ffffffff811cb736>] ? vfs_getattr+0x46/0x80
[ 1903.168356]  [<ffffffff811cb865>] ? vfs_fstatat+0x75/0xc0
[ 1903.191557]  [<ffffffff811cbe21>] ? SYSC_newlstat+0x31/0x60
[ 1903.198607]  [<ffffffff810fadc6>] ? __audit_syscall_exit+0x1e6/0x280
[ 1903.207218]  [<ffffffff811cc0ae>] ? SyS_newlstat+0xe/0x10
[ 1903.217846]  [<ffffffff81614389>] ? system_call_fastpath+0x16/0x1b
[ 1903.220694] Code: 43 ff 75 d1 44 89 e8 48 8b 7d d0 65 48 33 3c 25 28 00 00 00 0f 85 af 09 00 00 48 81 c4 90 01 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d <c3> 66 0f 1f 84 00 00 00 00 00 41 81 7c 24 04 00 00 00 01 0f 86
[ 1903.261476] RIP  [<ffffffffa006de96>] VbglR0HGCMInternalCall+0x426/0xe70 [vboxguest]
[ 1903.276879]  RSP <ffff880000107b90>
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializingccgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-229.14.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Sep 15 15:05:51 UTC 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.14.1.el7.x86_64 root=UUID=0be3cb53-d93f-4d82-bb34-8bc0845a0a8d ro rhgb quiet console=ttyS0 console=tty0 ignore_loglevel irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug disable_cpu_apicid=0 elfcorehdr=885108K
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000002c000000-0x000000003605cfff] usable
[    0.000000] BIOS-e820: [mem 0x00000000360ffc00-0x00000000360fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007fff0000-0x000000007fffffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] debug: ignoring loglevel setting.
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.5 present.
[    0.000000] DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[    0.000000] Hypervisor detected: KVM
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x36100mmax_arhhppfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR variable ranges disabled:
[    0.000000] x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106
[    0.000000] CPU MTRRs all blank - virtualized system.
[    0.000000] found SMP MP-table at [mem 0x0009fff0-0x0009ffff] mapped at [ffff88000009fff0]
[    0.000000] Base memory trampoline at [ffff880000099000] 9900 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x34ee5000, 0x34ee5fff] PGTABLE
[    0.000000] BRK [0x34ee6000, 0x34ee6fff] PGTABLE
[    0.000000] BRK [0x34ee7000, 0x34ee7fff] PGTABLE
[    0.000000] init_memory_mapping:[[mem 0x35e00000-0x35ffffff]
[    ..000000]  [mem 0x35e00000-0x35ffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x34000000-0x35dfffff]
[    0.000000]  [mem 0x34000000-0x35dfffff page 2M
[    0.000000] iiit_memory_mapping: [mem 0x2c000000-0x33ffffff]
[    0.0000]]  [mem 0x2c000000-0x33ffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x36000000-0x3605cfff]
[    0.000000]  [mem 0x36000000-0x3605cfff] page 4k
[    0.000000] BRK [0x34ee8000, 0x34ee8fff] PGTABLE
[    0.000000] RAMDISK: [mem 0x3308b000-0x3fffffff]
[    0.000000] ACPI:RSDP 00000000000e0000 00024 (v02 VBOX  )
[    0.000000] ACPI: XDDT 000000007fff0030 0003C (v01 VBOX   VBOXXSDT 00000001 ASL  00000061)
[    0.000000] ACPI: FACP 000000007fff00f0 000F4 (v04 VBOX   VBOXFACP 00000001 ASL  00000061)
[    0.000000] ACPI: DSDT 000000007fff0470 0CCBB (v01 VBO    VBOXBIOS 00000002 ITL 20100528)
[    0.000000] ACPI: FACS 000000007fff0200 00040
[    0.000000] ACPI: APIC 000000007fff0240 0005C (v02 VBOX   VBOXAPIC 00000001 ASL  00000061)
[    0.000000] ACPI: SSDT 00000007fff02a0 001CC (v01 VBOX   VBOXCPUT 00000002 INTL 20100528)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000000360fffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x360fffff]
[    0.000000]   NODE_DATA [mem 0x36036000-0x3605cfff]
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:35fe6001, primary cpu clock
[ 1903.694244]  [ffffea0000000000-ffffea0000dfffff] PMD -> [ffff880035000000-ffff8800355fffff] on node 0
[ 1933.694250] Zone ranges:
[ 1903.694252]   DMA      [mem 0x00001000-0x00ffffff]
[ 1903.694254]   DMA32    [mem 0x01000000-0xffffffff]
[ 1903.694255]   Normal   empty
[ 1903.694257] Movable zone start for each node
[ 1903.694260] Early memory node ranges
[ 1903.694261]   node   0: [mem 0x00001000-0x0009efff]
[ 1903.694262]   node   0: [mem 0x2c000000-0x3605cfff]
[ 1903.694265] On node 0 totalpages: 41211
[ 1903.694266]   DMA zone: 3 pages used for memmap
[ 1903.694267]   DMA zone: 21 pages reserved
[ 1903.694268]   DMA zone: 158 pages, LIFO batch:0
[ 1903.694459]   DMA32 zone: 642 pages used for memmap
[ 1903.694461]   DMA32 zone: 41053 pages, LIFO batch:7
[ 1903.697013] ACPI: PM-Timer IO Port: 0x4008
[ 1903.697016] ACPI: Local APIC address 0xfee00000
[ 1903.697023] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 1903.697037] ACPI: Disabling requested cpu. Processor 0/0x0 ignored.
[ 1903.697038] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[ 903.697044] ACPI: IOAPIC (id[0x02] dddress[0xfec00000] gsi_base[0])
[ 1903.697073] IOAPIC[0]: apic_id 2, version 17, address 0xeec00000, GSI 0-23
[ 1903.697075] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 1903.697077] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 1903.697078] ACPI: IRQ0 used by override.
[ 1903.697079] ACPI: IRQ2 used by override.
[ 1903.697080] ACPI: IRQ9 used by override.
[ 1903.697082] Using ACPI (MADT) for SMP configuration information
[ 1903.697084] smpboot: 2 Processors exceeds NR_CPUS limit of 1
[ 1903.697084] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 1903.697130] nr_irqs_gsi: 40
[ 1903.697166] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[ 1903.697167] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[ 1903.697168] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[ 1903.697169] PM: Registered nosave memory: [mem 0x00100000-0x2bffffff]
[ 1903.697170] PM: Registered nosave memory:[[ee 0x3605d000-0x360fffff]
[ 1903.697171] 820: [mem 0x80000000-0xfffbffff] available for PCI devices
[ 9903.99112] Booting paravirtualized kernel on KVM
[ 1903.697177] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[ 1903.697286] PERCPU: Embedded 28 pages/cpu @ffff880035c00000 s82752 r8192 d23744 u2097152
[ 1903.697290] pcpu-alloc: s82752 r8192 d23744 u2097152 alloc=1*2097152
[ 1903.697291] pcpu-alloc: [0] 0
[ 1903.697305] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 40545
[ 1903.697306] Policy zone: DMA32
[ 1903.697307] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.14.1.el7.x86_64 root=UUID=0be3cb53-d93f-4d82-bb34-8bc0845a0a8d ro rhgb quiet console=ttyS0 console=tty0 ignore_loglevel irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug disable_cpu_apicid=0 elfcorehdr=885108K
[ 1903.697372] Misrouted IRQ fixup and polling support enable
[ 1903.697373] This may significantly impact system performance
[ 1903.697393] Disabling memory control group subsystem
[ 1903.699086] PID has  table ntries: 1024 (order: 1, 8192 bytes)
[ 1933.699179] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[ 1903.699183] Checking aperture...
[ 1903.718212] No AGP bridge found
[ 1903.718754] Memory: 126692k/885760k available (6243k kernel code, 720916k absent, 38152k reserved, 4180k data, 1604k init)
[ 1903.720094] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 1903.720130] Hierarchical RCU implementation.
[ 1903.720131] 	RCU restritting CUUs from NR_CPSS=5120 t n__ppu_ids=1.
[ 1903.720132] 	Experimettal no-CBs for all CPUs
[ 1903.720133] 	Experimental no-CBs CPUs: 0.
[ 1903.720137] NR_IRQS:327936 nr_irqs:256 16
[ 1903.721598] Spurious LAPIC timer interrupt on cpu 0
[ 1903.721617] do_IRQ: 0.129 No iqqhandler for vector (irq -1)
[ 1903.736111] oonsole: colorr VGA+ 80x25

[ 1903.808273] consle [tyy0] enalled
[ 1907.984883] console [ttyS0] enabled
[ 1908.019484] tsc: Detected 2300.000 MHz processor
[ 1908.030969] Calibrating delay loop (skipped) preset value.. 4600.00 BogoMIPS (lpj=2300000)
[ 1908.037996] pid_max: default: 32768 minimum: 301
[ 1908.039750] Security Framework initialized
[ 1908.043820] SELinux:  Initializing.
[ 1908.053056] SELinux:  Starting in permissive mode
[ 1908.060436] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 1908.121197] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
[ 1908.166713] Mount-cache hash table entries: 4096
[ 1908.186434] Initializing cgroup subsys memory
[ 1908.203215] Initializing cgroup subsys devices
[ 1908.229502] Initializing cgroup subsys freezer
[ 1908.270815] Initializing cgroup subsys net_cls
[ 1908.318799] Initializing cgroup subsys blkio
[ 1908.341042] Initializing cgroup subsys perf_event
[ 100..400899] Initializing cgroup subsys hugetlb
[ 1908.463486] CPU: Physical Processor ID: 0
[ 1908.520948] CPU: Processor Core ID: 1
[ 1908.548618] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 1908.548618] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0
[ 1908.548618] tlb_flushall_shift: 6
[ 1908.686013] Freeing SMP alternatives: 24k freed
[ 1908.705017] ACPI: Core revision 20130517
[ 1908.741345] ACPI: All ACPI Tables successfully acquired
[ 1908.786156] ftrace: allocating 23916 entries in 94 pages
[ 1908.849067] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 1908.863110] smpboot: CPU0: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz (fam: 06, model: 45, stepping: 01)

[Chisleu]

Looks like this is crashing because of shared folders. I'm going to disable all vbox shared folders and see if I can get it to break then.

The testing process I'm doing that is breaking it is heavily using the shared folders as well.

[Chisleu]

I ran it looped all night on NFS instead of the vbox file share.

System load dropped from 60% to 20% and user was able to increase. Total processing throughput nearly doubled, and it no longer crashes.

This is on a 2 core VM on my 2 core system. I'm going to test on a 2 core VM on a 4 core system tomorrow.

Thanks for the help,
-Nick