Page 1 of 2
Virtualbox freezing host, requires power cycle to clear
Posted: 7. Jul 2016, 23:06
by Pop
Please forgive me in advance if I don't provide what is needed for effective recommendations. Though I have used Linux for a number of years for various tasks, I am no expert and rely a great deal on the kindness of strangers on multiple forums. And, I'm a brand-new user of Virtualbox. When it's working I'm loving it, so I'm anxious for this to be stable!
About every 6 hours my guests and hosts lock up, requiring a hard reset. It seems to start with the Joomla! guest (see below) not responding to any network traffic, progresses to the other 2 guests, then the host OS locks hard. If I catch it quickly enough I can sometimes prevent a complete lockup by killing each guest, then killing Virtualbox.
My hardware is an HP ML310e G8v2 with 1 quadcore processor, 16GB RAM, and 4 1-terabyte drives. Hard drive 1 is for the base OS (Debian Jessie with all patches up to date); drive 2 is unused; drive 3 is the repository for my guests, and drive 4 is temporarily my backup destination. No RAID at this time. Hyperthreading disabled.
Current version of virtualbox: 5.0.24. My troubles have occurred on 5.0.22 and 4.6.36 too.
I'm running 3 guests, all based on Debian Jessie. Intel network on all 3. Nested paging disabled, PAE/NX enabled, Guest additions installed on all 3. Execution cap set to 75% on all guests.
1. an admin terminal for general web browsing, libre office, and other conveniences. 1 processor, 1GB RAM, 100GB drive.
2. SuiteCRM from turnkeylinux. 1 processor, 2GB RAM, 250GB drive.
3. Joomla! 3 from Turnkeylinux. 1 processor, 2GB RAM, 100GB drive.
Bridged network. In BIOS network speed set explicitly at 100MB full duplex.
Any help would be appreciated.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 7. Jul 2016, 23:28
by Perryg
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 7. Jul 2016, 23:43
by Pop
Thanks Perry.
I think I provided everything outlined in your note except that the hosts and guests are all running 64-bit versions of Debian Jessie; and the log file. Log coming momentarily.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 7. Jul 2016, 23:51
by Pop
Log from the joomla guest:
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 00:04
by Perryg
Code: Select all
00:00:00.480586 AIOMgr: Endpoint for file '/media/mark/sdc/joomla/joomla.vmdk' (flags 000c0723) created successfully
00:00:00.752539 AIOMgr: Flush failed with VERR_INVALID_PARAMETER, disabling async flushes
Is this guest stored on a removable drive, and if so are the other guest also on the same drive?
Code: Select all
VirtualBox VM 5.0.24 r108355 linux.amd64 (Jun 28 2016 16:35:55) release log
00:00:17.880873 VMMDev: Guest Additions information report: Version 5.0.16 r105871 '5.0.16'
Guest additions are fairly old consider upgrading
Code: Select all
00:00:28.255549 VMMDev: Guest Log: 00:00:00.064911 vminfo rtldrNativeLoad: dlopen('libdbus-1.so.3', RTLD_NOW | RTLD_LOCAL) failed: libdbus-1.so.3: cannot open shared object file: No such file or directory
00:00:28.256702 VMMDev: Guest Log: 00:00:00.065810 vminfo Error: Unable to connect to system D-Bus (1/3): D-Bus not installed
00:00:28.293142 GUI: UISession::sltAdditionsChange: GA state really changed, notifying listeners.
00:00:28.293189 GUI: UIMachineViewNormal::adjustGuestScreenSize: Adjust guest-screen size if necessary.
00:00:28.293199 GUI: UISession::sltAdditionsChange: GA state change event came, notifying listeners.
00:00:28.293207 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=0, Machine-state=5
00:00:33.303539 VMMDev: Guest Log: 00:00:05.114481 vminfo Error: Unable to connect to system D-Bus (2/3): D-Bus not installed
00:00:38.312437 VMMDev: Guest Log: 00:00:10.123382 vminfo Error: Unable to connect to system D-Bus (3/3): D-Bus not installed
This concerns me as well.
Do the other guest have the same issues in the log file?
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 01:21
by Pop
guests are all on the same non-removable SATA drive.
As for the guest additions, isn't 5.0.16 the latest available? That's what I downloaded from
https://www.virtualbox.org/wiki/Downloads.
I'll check the logs for the other guests tomorrow, office now closed and I don't have remote access just yet.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 01:57
by Perryg
Could you also post the results of df -h from the host terminal?
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 14:31
by Pop
Logs from my other two guests are attached.
Output of df -h on host:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 886G 7.6G 834G 1% /
udev 10M 0 10M 0% /dev
tmpfs 3.2G 8.9M 3.2G 1% /run
tmpfs 7.9G 448K 7.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
tmpfs 1.6G 20K 1.6G 1% /run/user/1000
/dev/sr0 58M 58M 0 100% /media/cdrom0
/dev/sdc 917G 15G 856G 2% /media/mark/sdc
/dev/sdd 917G 51G 820G 6% /media/mark/sdd1
By the way, I've been running for over a day now without a lockup, but only the suitecrm guest is active. The other two guests are turned off.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 14:58
by Perryg
While I do see some errors in the logs you provide I do not see any that should cause a lock up on the host. I believe this is being triggered by a load of some kind as indicated by your reply that with a single guest you do not have a lock up. It probably will help to isolate this by starting another guest and see if it fails and if not then the third. You should monitor the processor usage during this time and report any unusual indications. While I can not tell you at this time that the guest/s are the real issue I am not fond of turnkey OSes. There can be a lot of differences having a guest built in a different environment than the one it is actually being used in.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 15:16
by Pop
Thanks Perry.
I assume that you are recommending that I monitor the physical processor in the host, correct?
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 15:22
by Perryg
Yes.
You can also monitor running processes and see if one of them spikes and if so why. I would also recommend watching dmesg and sys logs. Something must indicate what is happening and then figure out why.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 15:25
by Perryg
Oh and while this probably will not help or hinder the guest/s I find hyper-threading greatly improves the host run/build times. I see no reason to disable it unless this is a company thing.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 16:48
by Pop
Less than an hour after starting the joomla guest both it and suitecrm guest have both stopped responding. Can't ping, can't access from browser. I can ping the host, though.
When I try to get the logs from the virtualbox UI the OS responds with "virtualbox has stopped responding" and gives me the choice to wait or kill.
Here's the output of dmesg:
out.txt- (126.13 KiB) Downloaded 14 times
And syslog:
I'm looking through them now, but any guidance would be appreciated.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 16:57
by Perryg
This is what is happening.
Code: Select all
ul 8 09:32:26 vserver kernel: [148595.307485] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 2, t=5252 jiffies, g=405431, c=405430, q=11708)
Jul 8 09:32:26 vserver kernel: [148595.307504] sending NMI to all CPUs:
You say it happens shortly after the joomla guest starts so does it happen if you start the other guest instead? This would isolate it to the joomla guest as the cause but don't jump to the conclusion that is is the real reason yet.
Re: Virtualbox freezing host, requires power cycle to clear
Posted: 8. Jul 2016, 16:57
by Pop
Hmmm. Never noticed this happen before. Now the virtualbox UI and both guests are responding again.
From the end of the joomla log:
Code: Select all
00:00:19.704862 AC97: Reset
00:00:19.705367 AC97: Reset
00:00:27.482562 VMMDev: Guest Log: VBoxService 5.0.16 r105871 (verbosity: 0) linux.amd64 (Mar 4 2016 17:38:55) release log
00:00:27.482591 VMMDev: Guest Log: 00:00:00.000729 main Log opened 2016-07-08T13:20:34.451526000Z
00:00:27.482747 VMMDev: Guest Log: 00:00:00.000927 main OS Product: Linux
00:00:27.482788 VMMDev: Guest Log: 00:00:00.000974 main OS Release: 3.16.0-4-amd64
00:00:27.482841 VMMDev: Guest Log: 00:00:00.001027 main OS Version: #1 SMP Debian 3.16.7-ckt25-2+deb8u2 (2016-06-25)
00:00:27.482876 VMMDev: Guest Log: 00:00:00.001060 main Executable: /opt/VBoxGuestAdditions-5.0.16/sbin/VBoxService
00:00:27.482881 VMMDev: Guest Log: 00:00:00.001062 main Process ID: 668
00:00:27.482886 VMMDev: Guest Log: 00:00:00.001063 main Package type: LINUX_64BITS_GENERIC
00:00:27.485419 VMMDev: Guest Log: 00:00:00.003526 main 5.0.16 r105871 started. Verbose level = 0
00:00:27.536425 GUI: UISession::sltAdditionsChange: GA state change event came, notifying listeners.
00:00:27.536451 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=0, Machine-state=5
00:00:27.564276 VMMDev: Guest Log: 00:00:00.082116 vminfo rtldrNativeLoad: dlopen('libdbus-1.so.3', RTLD_NOW | RTLD_LOCAL) failed: libdbus-1.so.3: cannot open shared object file: No such file or directory
00:00:27.566106 VMMDev: Guest Log: 00:00:00.084473 vminfo Error: Unable to connect to system D-Bus (1/3): D-Bus not installed
00:00:27.573409 GUI: UISession::sltAdditionsChange: GA state really changed, notifying listeners.
00:00:27.573433 GUI: UIMachineViewNormal::adjustGuestScreenSize: Adjust guest-screen size if necessary.
00:00:27.573440 GUI: UISession::sltAdditionsChange: GA state change event came, notifying listeners.
00:00:27.573445 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=0, Machine-state=5
00:00:32.573113 VMMDev: Guest Log: 00:00:05.093069 vminfo Error: Unable to connect to system D-Bus (2/3): D-Bus not installed
00:00:37.594480 VMMDev: Guest Log: 00:00:10.115431 vminfo Error: Unable to connect to system D-Bus (3/3): D-Bus not installed
00:03:07.609830 GUI: UIVMInfoDialog: Geometry loaded to: Origin=812x412, Size=600x450
00:04:32.486036 GUI: UIVMInfoDialog: Geometry saved as: Origin=812x412, Size=600x450
00:05:23.125993 GUI: UIMediumEnumerator: Medium-enumeration started...
00:05:23.208034 GUI: UIMediumEnumerator: Medium-enumeration finished!
01:30:34.038549 TM: Giving up catch-up attempt at a 4 149 335 229 410 ns lag; new total: 4 149 335 229 410 ns
01:30:34.038592 VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4151 seconds ago
01:30:34.039735 VMMDev: GuestHeartBeat: Guest is alive (gone 4 151 336 611 237 ns)
from the end of the suitecrm log:
00:00:01.404306 ******************** End of CPUID dump **********************
00:00:01.411213 VMEmt: Halt method global1 (5)
00:00:01.411248 VMEmt: HaltedGlobal1 config: cNsSpinBlockThresholdCfg=2000
00:00:01.411262 Changing the VM state from 'CREATING' to 'CREATED'
00:00:01.415316 Changing the VM state from 'CREATED' to 'POWERING_ON'
00:00:01.415513 AIOMgr: Endpoints without assigned bandwidth groups:
00:00:01.415523 AIOMgr: /media/mark/sdc/suitecrm/suitecrm.vmdk
00:00:01.415655 Changing the VM state from 'POWERING_ON' to 'RUNNING'
00:00:01.415669 Console: Machine state changed to 'Running'
00:00:01.421434 VMMDev: Guest Log: BIOS: VirtualBox 5.0.24
00:00:01.425814 PIT: mode=2 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:00:01.449846 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={7303a66d-433b-25a4-f9a8-fcadf87e0c2a} aComponent={DisplayWrap} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false aResultDetail=0
00:00:01.449913 Display::handleDisplayResize: uScreenId=0 pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0 flags=0x1
00:00:01.450034 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=720x400, Sending to async-handler
00:00:01.450105 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=720x400
00:00:01.450134 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=720x400
00:00:01.450145 GUI: UIFrameBufferPrivate::performResize: Size=720x400, Directly using source bitmap content
00:00:01.466273 PIIX3 ATA: Ctl#1: RESET, DevSel=0 AIOIf=0 CmdIf0=0x00 (-1 usec ago) CmdIf1=0x00 (-1 usec ago)
00:00:01.466368 PIIX3 ATA: Ctl#1: finished processing RESET
00:00:01.493825 AHCI#0: Reset the HBA
00:00:01.493983 AHCI#0: Port 0 reset
00:00:01.495559 VMMDev: Guest Log: BIOS: AHCI 0-P#0: PCHS=16383/16/63 LCHS=1024/255/63 0x000000001f400000 sectors
00:00:01.499359 PIT: mode=2 count=0x48d3 (18643) - 64.00 Hz (ch=0)
00:00:01.519116 Display::handleDisplayResize: uScreenId=0 pvVRAM=00007fbbb9ffe000 w=640 h=480 bpp=32 cbLine=0xA00 flags=0x1
00:00:01.519338 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=640x480, Sending to async-handler
00:00:01.519825 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=640x480
00:00:01.519922 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=640x480
00:00:01.520523 GUI: UIFrameBufferPrivate::performResize: Size=640x480, Directly using source bitmap content
00:00:01.723219 GUI: UIMachineViewNormal::resendSizeHint: Restoring guest size-hint for screen 0 to 720x400
00:00:01.723219 VMMDev: SetVideoModeHint: Got a video mode hint (720x400x32)@(0x0),(1;0) at 0
00:00:01.723219 GUI: 2D video acceleration is disabled
00:00:01.723219 GUI: HID LEDs sync is not supported on this platform
00:00:01.723219 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=0, Machine-state=5
00:00:04.024632 Display::handleDisplayResize: uScreenId=0 pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0 flags=0x1
00:00:04.024770 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=720x400, Sending to async-handler
00:00:04.025372 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=720x400
00:00:04.025635 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=720x400
00:00:04.025711 GUI: UIFrameBufferPrivate::performResize: Size=720x400, Directly using source bitmap content
00:00:04.043854 PIT: mode=2 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:00:04.044496 VMMDev: Guest Log: BIOS: Boot : bseqnr=1, bootseq=0231
00:00:04.044994 VMMDev: Guest Log: BIOS: Boot from Floppy 0 failed
00:00:04.045527 VMMDev: Guest Log: BIOS: Boot : bseqnr=2, bootseq=0023
00:00:04.054837 VMMDev: Guest Log: BIOS: CDROM boot failure code : 0004
00:00:04.055686 VMMDev: Guest Log: BIOS: Boot from CD-ROM failed
00:00:04.057681 VMMDev: Guest Log: BIOS: Boot : bseqnr=3, bootseq=0002
00:00:04.063370 VMMDev: Guest Log: BIOS: Booting from Hard Disk...
00:00:06.677355 PIT: mode=2 count=0x12a5 (4773) - 249.98 Hz (ch=0)
00:00:07.207153 PIT: mode=0 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:00:07.527385 OHCI: Software reset
00:00:08.631408 AHCI#0: Reset the HBA
00:00:08.711243 AHCI#0: Port 0 reset
00:00:08.758354 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0x00 (-1 usec ago) CmdIf1=0x00 (-1 usec ago)
00:00:08.758454 PIIX3 ATA: Ctl#0: finished processing RESET
00:00:08.761270 PIIX3 ATA: Ctl#1: RESET, DevSel=0 AIOIf=0 CmdIf0=0xa0 (-1 usec ago) CmdIf1=0x00 (-1 usec ago)
00:00:08.761334 PIIX3 ATA: Ctl#1: finished processing RESET
00:00:09.408831 OHCI: USB Reset
00:00:09.463596 OHCI: Software reset
00:00:09.463708 OHCI: USB Operational
00:00:09.580129 OHCI: USB Suspended
00:32:18.709360 TM: Not bothering to attempt catching up a 209 303 106 761 ns lag; new total: 209 303 106 761
41:07:45.729039 TM: Giving up catch-up attempt at a 60 002 162 715 ns lag; new total: 269 305 269 476 ns
41:10:17.986686 TM: Giving up catch-up attempt at a 60 061 575 889 ns lag; new total: 329 366 845 365 ns
42:23:48.148341 TM: Giving up catch-up attempt at a 4 409 375 243 734 ns lag; new total: 4 738 742 089 099 ns