Intermittent Hang/Crash
-
- Posts: 6
- Joined: 16. Dec 2010, 16:55
- Primary OS: Ubuntu 12.04
- VBox Version: OSE Debian
- Guest OSses: linux
Intermittent Hang/Crash
Hi all!
On an Ubuntu 14.04.3 LTS host I'm running some headless guest vm. One is an old debian guest that is running fine. Another one is an Ubuntu 14.04.3 LTS Server with ispconfig3 installed. Both are Vanilla. Just the standard install without anything special. I played around with the settings of Enable VT-x/AMD-V , Enable PAE/NX, Enable I/O APIC, Enable EFI (special OSes only) in phpvirtualbox without any luck. Most of the time it boots ok but at some point it crash. Looks like it's always the same thing. The vm just becomes unresponsive. The RDP is frozen like any other service on the vm.
In the vbox.log here is the repeating line every time.
17:00:17.273499 TM: Giving up catch-up attempt at a 60 001 253 009 ns lag; new total: 840 247 727 153 ns
The host is a dell 2950 2 x e5440 4 core 32 gig ecc ram.
I'm kind of lost here. Any help on this would be appreciated.
On an Ubuntu 14.04.3 LTS host I'm running some headless guest vm. One is an old debian guest that is running fine. Another one is an Ubuntu 14.04.3 LTS Server with ispconfig3 installed. Both are Vanilla. Just the standard install without anything special. I played around with the settings of Enable VT-x/AMD-V , Enable PAE/NX, Enable I/O APIC, Enable EFI (special OSes only) in phpvirtualbox without any luck. Most of the time it boots ok but at some point it crash. Looks like it's always the same thing. The vm just becomes unresponsive. The RDP is frozen like any other service on the vm.
In the vbox.log here is the repeating line every time.
17:00:17.273499 TM: Giving up catch-up attempt at a 60 001 253 009 ns lag; new total: 840 247 727 153 ns
The host is a dell 2950 2 x e5440 4 core 32 gig ecc ram.
I'm kind of lost here. Any help on this would be appreciated.
Last edited by spazio on 9. Dec 2015, 16:40, edited 1 time in total.
-
- Site Moderator
- Posts: 34369
- Joined: 6. Sep 2008, 22:55
- Primary OS: Linux other
- VBox Version: OSE self-compiled
- Guest OSses: *NIX
Re: Intermittent Hang/Crash
VirtualBox VM 5.0.10 r104061 linux.amd64 (Nov 10 2015 17:52:21) release log
00:00:00.366303 Oracle VM VirtualBox Extension Pack (Version: 5.0.4 r102546; VRDE Module: VBoxVRDP)
I would first update the extension pack to match the running version of VirtualBox.
00:00:00.366303 Oracle VM VirtualBox Extension Pack (Version: 5.0.4 r102546; VRDE Module: VBoxVRDP)
I would first update the extension pack to match the running version of VirtualBox.
-
- Posts: 6
- Joined: 16. Dec 2010, 16:55
- Primary OS: Ubuntu 12.04
- VBox Version: OSE Debian
- Guest OSses: linux
Re: Intermittent Hang/Crash
I missed that ! %?&*(*&
Thanks for pointing it!
I'll match the pack version and see.
Thanks again
Spazio
Thanks for pointing it!
I'll match the pack version and see.
Thanks again
Spazio
Re: Intermittent Hang/Crash
I am having the same issue. I have a small box which runs a dual core atom processor. It has 2GB of ram. It runs a pair of Openwrt images with 100% reliability. They run a 3.18. kernel. Then, I have a guest which runs Ubuntu 14.04.3 LTS X64. The guest just becomes non responsive (VRDE console & network pinging) after some period of time. It never recovers once it stops responding. It appears that the more load I push through the guest, the quicker this happens. I have been tracking the same issue for the last 4 or 5 releases at least and I suspect it goes back further, but I can't be sure. I was really hoping those last Atom fixes in the last release were the answer but apparently not. It the attached log the machine gave up after 2 hours or so. This is very reproducible at the moment as I'm trying to pull 1.5tb out of the guest and on to a backup hard drive. Oh, and when the machine is hung it powers off just fine, oddly enough.
- Attachments
-
- saved.log
- (98.57 KiB) Downloaded 102 times
-
- Site Moderator
- Posts: 34369
- Joined: 6. Sep 2008, 22:55
- Primary OS: Linux other
- VBox Version: OSE self-compiled
- Guest OSses: *NIX
Re: Intermittent Hang/Crash
VirtualBox VM 5.0.10 r104061 linux.amd64 (Nov 10 2015 17:52:21) release logBruksee wrote:I am having the same issue. I have a small box which runs a dual core atom processor. It has 2GB of ram. It runs a pair of Openwrt images with 100% reliability. They run a 3.18. kernel. Then, I have a guest which runs Ubuntu 14.04.3 LTS X64. The guest just becomes non responsive (VRDE console & network pinging) after some period of time. It never recovers once it stops responding. It appears that the more load I push through the guest, the quicker this happens. I have been tracking the same issue for the last 4 or 5 releases at least and I suspect it goes back further, but I can't be sure. I was really hoping those last Atom fixes in the last release were the answer but apparently not. It the attached log the machine gave up after 2 hours or so. This is very reproducible at the moment as I'm trying to pull 1.5tb out of the guest and on to a backup hard drive. Oh, and when the machine is hung it powers off just fine, oddly enough.
00:00:28.502596 VMMDev: Guest Additions information report: Version 4.3.22 r98236 '4.3.22_Ubuntu'
Not only are the guest additions out of date it looks like you are using the Ubuntu version of the additions instead of the ones from virtualbox. If so then mixing the official version with the ubuntu fork version of the additions has been known to cause issues. You would be better off removing the guest additions if you do not want to use the right ones.
Also not discounting the Atom processor as part of the problem as there have been a lot of reports that they do not work well with the added weight of running a virtual guest and some compatibility issues.
Re: Intermittent Hang/Crash
Gosh, what a dope. I will fix that and see how it goes. I could of sworn I had this all straightened out. I mean I would of bet $100 I had the right version.Perryg wrote:Bruksee wrote:I
Not only are the guest additions out of date it looks like you are using the Ubuntu version of the additions instead of the ones from virtualbox. If so then mixing the official version with the ubuntu fork version of the additions has been known to cause issues. You would be better off removing the guest additions if you do not want to use the right ones.
Also not discounting the Atom processor as part of the problem as there have been a lot of reports that they do not work well with the added weight of running a virtual guest and some compatibility issues.
EDIT: Yes, I had 5.0.10 installed but evidently the Ubuntu ones are hanging around. I will see if I can ditch those.
Code: Select all
[ 38.515363] 00:00:00.001829 main Executable: /opt/VBoxGuestAdditions-5.0.10/sbin/VBoxService
Re: Intermittent Hang/Crash
OK, now from the host:
From the guest:
So I do NOT know what happened last time I ran that. It clearly ran to some degree since I had the 5.0.10 daemon running, but seemingly it didn't manage to build and insert the kernel modules. Fingers crossed. Thanks for the pointer.
Code: Select all
01:24:53.551822 VMMDev: Guest Additions information report: Version 5.0.10 r104061 '5.0.10'
Code: Select all
[ 5093.216605] vboxguest: misc device minor 56, IRQ 20, I/O port d020, MMIO at 00000000f0000000 (size 0x400000)
[ 5093.216671] vboxguest: Successfully loaded version 5.0.10 (interface 0x00010004)
[ 5102.508594] vboxsf: Successfully loaded version 5.0.10 (interface 0x00010004)
-
- Site Moderator
- Posts: 34369
- Joined: 6. Sep 2008, 22:55
- Primary OS: Linux other
- VBox Version: OSE self-compiled
- Guest OSses: *NIX
Re: Intermittent Hang/Crash
One last tip.
If you did not purge the additions that were installed before you installed the official version you may still have an issue.
If you did not purge the additions that were installed before you installed the official version you may still have an issue.
Re: Intermittent Hang/Crash
I've had a good look around the system and it's now loading the kernel modules out of /lib/modules/3.19.0-30-generic/misc instead of /lib/modules/3.19.0-30-generic/kernel/ubuntu/vbox/ where they were previously. Those modules appear to have been deleted this time. Thanks again. I've just kicked off another rsync over ssh which has been inducing the issue reliably every couple of hours, so if that is still running in the morning, I'll be pretty confident in calling it a fix.Perryg wrote:One last tip.
If you did not purge the additions that were installed before you installed the official version you may still have an issue.
-
- Posts: 6
- Joined: 16. Dec 2010, 16:55
- Primary OS: Ubuntu 12.04
- VBox Version: OSE Debian
- Guest OSses: linux
Re: Intermittent Hang/Crash
Well I'm glad someone fix his problem!
As for my problem after de-installing the wrong ext packt and putting the right one, something new came up!
The host and the first vm are running fine but I still have some hang problem with the ubuntu 14 lts vm. This morning I was able to log after it was running for 15 hours without completely crashing ( big improvement ) but when I did apt-get update and upgrade. It hang and I got this on the consol:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
And in the vbox log this line still appear:
16:00:16.322610 TM: Giving up catch-up attempt at a 60 003 002 846 ns lag; new total: 20 341 059 990 371 ns
Init 0 didn't work or any other commands, I had to power off.
Any idea?
Spazio
As for my problem after de-installing the wrong ext packt and putting the right one, something new came up!
The host and the first vm are running fine but I still have some hang problem with the ubuntu 14 lts vm. This morning I was able to log after it was running for 15 hours without completely crashing ( big improvement ) but when I did apt-get update and upgrade. It hang and I got this on the consol:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
And in the vbox log this line still appear:
16:00:16.322610 TM: Giving up catch-up attempt at a 60 003 002 846 ns lag; new total: 20 341 059 990 371 ns
Init 0 didn't work or any other commands, I had to power off.
Any idea?
Spazio
- Attachments
-
- 20151209-0926VBox.log
- VBox.Log
- (111.52 KiB) Downloaded 17 times
-
- Site Moderator
- Posts: 34369
- Joined: 6. Sep 2008, 22:55
- Primary OS: Linux other
- VBox Version: OSE self-compiled
- Guest OSses: *NIX
Re: Intermittent Hang/Crash
@spazio,
TM is referring to time management and the fact that is can not refresh is only an indicator of the problem.
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! however is an indicator of something that is a real problem. See the following link and see if something there is a cause.
https://www.google.com/search?q=NMI+wat ... gws_rd=ssl
TM is referring to time management and the fact that is can not refresh is only an indicator of the problem.
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! however is an indicator of something that is a real problem. See the following link and see if something there is a cause.
https://www.google.com/search?q=NMI+wat ... gws_rd=ssl
Re: Intermittent Hang/Crash
Well mine is not fixed. Crashed twice now in the same way. I am running this all headless, without the gui. Is there an concise idiots guide to positioning myself to collect/report something that will meaningfully advance some investigation. Most of what I've found seems to assume a GUI is present.
EDIT: as a starting point I have enabled a virtual serial console for the problematic virtual machine and got the console going out of it now. The next time it hangs it will be interesting to see if there's anything on the serial console... ?
Code: Select all
00:00:35.494357 VMMDev: Guest Log: 00:00:00.001193 main OS Release: 3.19.0-39-generic
00:00:35.494454 VMMDev: Guest Log: 00:00:00.001291 main OS Version: #44~14.04.1-Ubuntu SMP Wed Dec 2 10:00:35 UTC 2015
00:00:35.494557 VMMDev: Guest Log: 00:00:00.001389 main Executable: /opt/VBoxGuestAdditions-5.0.10/sbin/VBoxService
00:00:35.494608 VMMDev: Guest Log: 00:00:00.001393 main Process ID: 1204
00:00:35.494624 VMMDev: Guest Log: 00:00:00.001396 main Package type: LINUX_64BITS_GENERIC
00:00:35.511311 VMMDev: Guest Log: 00:00:00.018085 main 5.0.10 r104061 started. Verbose level = 0
04:32:15.399430 VMMDev: vmmDevHeartbeatFlatlinedTimer: Guest seems to be unresponsive. Last heartbeat received 4 seconds ago
Re: Intermittent Hang/Crash
Nothing on the serial prompt.Bruksee wrote:
EDIT: as a starting point I have enabled a virtual serial console for the problematic virtual machine and got the console going out of it now. The next time it hangs it will be interesting to see if there's anything on the serial console... ?
Re: Intermittent Hang/Crash
Well I spun on a Debian 8 image and connected my large volume to it, set it up just like the Ubuntu image, and that thing is rock solid. No crashes at all for 18 hours so far, and a constant 100% cpu load the entire time. So something to do with Ubuntu
-
- Posts: 6
- Joined: 16. Dec 2010, 16:55
- Primary OS: Ubuntu 12.04
- VBox Version: OSE Debian
- Guest OSses: linux
Re: Intermittent Hang/Crash
After soooo much reading and changing configs I was able to get the vm up without crashing for over 5 days now. Nothing wrong in the log.
For others here is my solution:
Looks like the virtualbox version 5 is quite different from the version 4 regarding to ubuntu 14.04 vanilla. I mean out of the box I'm using 14.04 on version 4. Here is the links I'm referring to:
http://askubuntu.com/questions/687610/h ... 4-04-guest
I changed the generic ubuntu kernel for the virtual
viewtopic.php?f=3&p=185794 but after installation read carefully because it's not label as virtual but still named generic:
http://askubuntu.com/questions/503020/l ... -04-trusty
And finally I added grub arguments on the kernel clock=pit nosmp noapic nolapic
explanation :
The “clock=pit” prevents the clock from running too quickly and “nosmp noapic nolapic” prevents the clock from running too slowly. Remember you must reboot your machine for this change to take effect.
from :
http://www.excaliburtech.net/archives/2
So in retrospect looks like I had a clock problem and the wrong kernel headers with the version 5 of virtualbox new features.
Anyways If it crash again I'll keep you posted.
So long so good!
Thanks
Spazio
For others here is my solution:
Looks like the virtualbox version 5 is quite different from the version 4 regarding to ubuntu 14.04 vanilla. I mean out of the box I'm using 14.04 on version 4. Here is the links I'm referring to:
http://askubuntu.com/questions/687610/h ... 4-04-guest
I changed the generic ubuntu kernel for the virtual
viewtopic.php?f=3&p=185794 but after installation read carefully because it's not label as virtual but still named generic:
http://askubuntu.com/questions/503020/l ... -04-trusty
And finally I added grub arguments on the kernel clock=pit nosmp noapic nolapic
explanation :
The “clock=pit” prevents the clock from running too quickly and “nosmp noapic nolapic” prevents the clock from running too slowly. Remember you must reboot your machine for this change to take effect.
from :
http://www.excaliburtech.net/archives/2
So in retrospect looks like I had a clock problem and the wrong kernel headers with the version 5 of virtualbox new features.
Anyways If it crash again I'll keep you posted.
So long so good!
Thanks
Spazio