Intermittent Hang/Crash

Discussions related to using VirtualBox on Linux hosts.
spazio
Posts: 6
Joined: 16. Dec 2010, 16:55
Primary OS: Ubuntu 12.04
VBox Version: OSE Debian
Guest OSses: linux

Intermittent Hang/Crash

Post by spazio »

Hi all!

On an Ubuntu 14.04.3 LTS host I'm running some headless guest vm. One is an old debian guest that is running fine. Another one is an Ubuntu 14.04.3 LTS Server with ispconfig3 installed. Both are Vanilla. Just the standard install without anything special. I played around with the settings of Enable VT-x/AMD-V , Enable PAE/NX, Enable I/O APIC, Enable EFI (special OSes only) in phpvirtualbox without any luck. Most of the time it boots ok but at some point it crash. Looks like it's always the same thing. The vm just becomes unresponsive. The RDP is frozen like any other service on the vm.
In the vbox.log here is the repeating line every time.

17:00:17.273499 TM: Giving up catch-up attempt at a 60 001 253 009 ns lag; new total: 840 247 727 153 ns

The host is a dell 2950 2 x e5440 4 core 32 gig ecc ram.

I'm kind of lost here. Any help on this would be appreciated.
Last edited by spazio on 9. Dec 2015, 16:40, edited 1 time in total.
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Intermittent Hang/Crash

Post by Perryg »

VirtualBox VM 5.0.10 r104061 linux.amd64 (Nov 10 2015 17:52:21) release log
00:00:00.366303 Oracle VM VirtualBox Extension Pack (Version: 5.0.4 r102546; VRDE Module: VBoxVRDP)

I would first update the extension pack to match the running version of VirtualBox.
spazio
Posts: 6
Joined: 16. Dec 2010, 16:55
Primary OS: Ubuntu 12.04
VBox Version: OSE Debian
Guest OSses: linux

Re: Intermittent Hang/Crash

Post by spazio »

I missed that ! %?&*(*&
Thanks for pointing it!
I'll match the pack version and see.

Thanks again
Spazio
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

I am having the same issue. I have a small box which runs a dual core atom processor. It has 2GB of ram. It runs a pair of Openwrt images with 100% reliability. They run a 3.18. kernel. Then, I have a guest which runs Ubuntu 14.04.3 LTS X64. The guest just becomes non responsive (VRDE console & network pinging) after some period of time. It never recovers once it stops responding. It appears that the more load I push through the guest, the quicker this happens. I have been tracking the same issue for the last 4 or 5 releases at least and I suspect it goes back further, but I can't be sure. I was really hoping those last Atom fixes in the last release were the answer but apparently not. It the attached log the machine gave up after 2 hours or so. This is very reproducible at the moment as I'm trying to pull 1.5tb out of the guest and on to a backup hard drive. Oh, and when the machine is hung it powers off just fine, oddly enough.
Attachments
saved.log
(98.57 KiB) Downloaded 91 times
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Intermittent Hang/Crash

Post by Perryg »

Bruksee wrote:I am having the same issue. I have a small box which runs a dual core atom processor. It has 2GB of ram. It runs a pair of Openwrt images with 100% reliability. They run a 3.18. kernel. Then, I have a guest which runs Ubuntu 14.04.3 LTS X64. The guest just becomes non responsive (VRDE console & network pinging) after some period of time. It never recovers once it stops responding. It appears that the more load I push through the guest, the quicker this happens. I have been tracking the same issue for the last 4 or 5 releases at least and I suspect it goes back further, but I can't be sure. I was really hoping those last Atom fixes in the last release were the answer but apparently not. It the attached log the machine gave up after 2 hours or so. This is very reproducible at the moment as I'm trying to pull 1.5tb out of the guest and on to a backup hard drive. Oh, and when the machine is hung it powers off just fine, oddly enough.
VirtualBox VM 5.0.10 r104061 linux.amd64 (Nov 10 2015 17:52:21) release log
00:00:28.502596 VMMDev: Guest Additions information report: Version 4.3.22 r98236 '4.3.22_Ubuntu'

Not only are the guest additions out of date it looks like you are using the Ubuntu version of the additions instead of the ones from virtualbox. If so then mixing the official version with the ubuntu fork version of the additions has been known to cause issues. You would be better off removing the guest additions if you do not want to use the right ones.

Also not discounting the Atom processor as part of the problem as there have been a lot of reports that they do not work well with the added weight of running a virtual guest and some compatibility issues.
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

Perryg wrote:
Bruksee wrote:I
Not only are the guest additions out of date it looks like you are using the Ubuntu version of the additions instead of the ones from virtualbox. If so then mixing the official version with the ubuntu fork version of the additions has been known to cause issues. You would be better off removing the guest additions if you do not want to use the right ones.

Also not discounting the Atom processor as part of the problem as there have been a lot of reports that they do not work well with the added weight of running a virtual guest and some compatibility issues.
Gosh, what a dope. I will fix that and see how it goes. I could of sworn I had this all straightened out. I mean I would of bet $100 I had the right version. :oops:

EDIT: Yes, I had 5.0.10 installed but evidently the Ubuntu ones are hanging around. I will see if I can ditch those.

Code: Select all

[   38.515363] 00:00:00.001829 main     Executable: /opt/VBoxGuestAdditions-5.0.10/sbin/VBoxService
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

OK, now from the host:

Code: Select all

01:24:53.551822 VMMDev: Guest Additions information report: Version 5.0.10 r104061 '5.0.10'
From the guest:

Code: Select all

[ 5093.216605] vboxguest: misc device minor 56, IRQ 20, I/O port d020, MMIO at 00000000f0000000 (size 0x400000)
[ 5093.216671] vboxguest: Successfully loaded version 5.0.10 (interface 0x00010004)
[ 5102.508594] vboxsf: Successfully loaded version 5.0.10 (interface 0x00010004)
So I do NOT know what happened last time I ran that. It clearly ran to some degree since I had the 5.0.10 daemon running, but seemingly it didn't manage to build and insert the kernel modules. Fingers crossed. Thanks for the pointer.
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Intermittent Hang/Crash

Post by Perryg »

One last tip.

If you did not purge the additions that were installed before you installed the official version you may still have an issue.
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

Perryg wrote:One last tip.

If you did not purge the additions that were installed before you installed the official version you may still have an issue.
I've had a good look around the system and it's now loading the kernel modules out of /lib/modules/3.19.0-30-generic/misc instead of /lib/modules/3.19.0-30-generic/kernel/ubuntu/vbox/ where they were previously. Those modules appear to have been deleted this time. Thanks again. I've just kicked off another rsync over ssh which has been inducing the issue reliably every couple of hours, so if that is still running in the morning, I'll be pretty confident in calling it a fix. :)
spazio
Posts: 6
Joined: 16. Dec 2010, 16:55
Primary OS: Ubuntu 12.04
VBox Version: OSE Debian
Guest OSses: linux

Re: Intermittent Hang/Crash

Post by spazio »

Well I'm glad someone fix his problem!

As for my problem after de-installing the wrong ext packt and putting the right one, something new came up!
The host and the first vm are running fine but I still have some hang problem with the ubuntu 14 lts vm. This morning I was able to log after it was running for 15 hours without completely crashing ( big improvement ) but when I did apt-get update and upgrade. It hang and I got this on the consol:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
And in the vbox log this line still appear:
16:00:16.322610 TM: Giving up catch-up attempt at a 60 003 002 846 ns lag; new total: 20 341 059 990 371 ns
Init 0 didn't work or any other commands, I had to power off.

Any idea?
Spazio
Attachments
20151209-0926VBox.log
VBox.Log
(111.52 KiB) Downloaded 11 times
Perryg
Site Moderator
Posts: 34369
Joined: 6. Sep 2008, 22:55
Primary OS: Linux other
VBox Version: OSE self-compiled
Guest OSses: *NIX

Re: Intermittent Hang/Crash

Post by Perryg »

@spazio,

TM is referring to time management and the fact that is can not refresh is only an indicator of the problem.
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! however is an indicator of something that is a real problem. See the following link and see if something there is a cause.
https://www.google.com/search?q=NMI+wat ... gws_rd=ssl
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

Well mine is not fixed. Crashed twice now in the same way. I am running this all headless, without the gui. Is there an concise idiots guide to positioning myself to collect/report something that will meaningfully advance some investigation. Most of what I've found seems to assume a GUI is present.

Code: Select all

00:00:35.494357 VMMDev: Guest Log: 00:00:00.001193 main     OS Release: 3.19.0-39-generic
00:00:35.494454 VMMDev: Guest Log: 00:00:00.001291 main     OS Version: #44~14.04.1-Ubuntu SMP Wed Dec 2 10:00:35 UTC 2015
00:00:35.494557 VMMDev: Guest Log: 00:00:00.001389 main     Executable: /opt/VBoxGuestAdditions-5.0.10/sbin/VBoxService
00:00:35.494608 VMMDev: Guest Log: 00:00:00.001393 main     Process ID: 1204
00:00:35.494624 VMMDev: Guest Log: 00:00:00.001396 main     Package type: LINUX_64BITS_GENERIC
00:00:35.511311 VMMDev: Guest Log: 00:00:00.018085 main     5.0.10 r104061 started. Verbose level = 0
04:32:15.399430 VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago
EDIT: as a starting point I have enabled a virtual serial console for the problematic virtual machine and got the console going out of it now. The next time it hangs it will be interesting to see if there's anything on the serial console... ?
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

Bruksee wrote:
EDIT: as a starting point I have enabled a virtual serial console for the problematic virtual machine and got the console going out of it now. The next time it hangs it will be interesting to see if there's anything on the serial console... ?
Nothing on the serial prompt. :|
Bruksee
Posts: 8
Joined: 9. Dec 2015, 01:27

Re: Intermittent Hang/Crash

Post by Bruksee »

Well I spun on a Debian 8 image and connected my large volume to it, set it up just like the Ubuntu image, and that thing is rock solid. No crashes at all for 18 hours so far, and a constant 100% cpu load the entire time. So something to do with Ubuntu :?
spazio
Posts: 6
Joined: 16. Dec 2010, 16:55
Primary OS: Ubuntu 12.04
VBox Version: OSE Debian
Guest OSses: linux

Re: Intermittent Hang/Crash

Post by spazio »

After soooo much reading and changing configs I was able to get the vm up without crashing for over 5 days now. Nothing wrong in the log.

For others here is my solution:

Looks like the virtualbox version 5 is quite different from the version 4 regarding to ubuntu 14.04 vanilla. I mean out of the box I'm using 14.04 on version 4. Here is the links I'm referring to:

http://askubuntu.com/questions/687610/h ... 4-04-guest

I changed the generic ubuntu kernel for the virtual
viewtopic.php?f=3&p=185794 but after installation read carefully because it's not label as virtual but still named generic:
http://askubuntu.com/questions/503020/l ... -04-trusty

And finally I added grub arguments on the kernel clock=pit nosmp noapic nolapic

explanation :
The “clock=pit” prevents the clock from running too quickly and “nosmp noapic nolapic” prevents the clock from running too slowly. Remember you must reboot your machine for this change to take effect.
from :
http://www.excaliburtech.net/archives/2

So in retrospect looks like I had a clock problem and the wrong kernel headers with the version 5 of virtualbox new features.
Anyways If it crash again I'll keep you posted.
So long so good!

Thanks
Spazio
Post Reply