A running guest seems to be stuck

Discussions related to using VirtualBox on Linux hosts.
Post Reply
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

A running guest seems to be stuck

Post by rbarak »

I have a Guest that is not answering (even to ping)

Code: Select all

$ ping -c1 main-direct-ingest-vb-maor.local
PING main-direct-ingest.local (192.168.1.76) 56(84) bytes of data.
From virtualbox01.local (192.168.1.140) icmp_seq=1 Destination Host Unreachable

--- main-direct-ingest.local ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
but still shows in the running list

Code: Select all

$ vboxmanage list runningvms  | grep main-direct-ingest
"main-direct-ingest-vb-maor" {eaf48de3-2eca-47a8-9014-b286c5a5eb99}
The last lines in the VBox.log are:

Code: Select all

00:01:17.940845 VMMDev: Guest Log: 00:00:00.000373 main     Process ID: 5405
00:01:17.940848 VMMDev: Guest Log: 00:00:00.000373 main     Package type: LINUX_64BITS_GENERIC
00:01:17.942398 VMMDev: Guest Log: 00:00:00.001923 main     5.2.20 r125813 started. Verbose level = 0
00:01:24.612349 Display::i_handleDisplayResize: uScreenId=0 pvVRAM=00007f1697000000 w=1024 h=768 bpp=32 cbLine=0x1000 flags=0x0 origin=0,0
51:23:16.479747 AHCI#0: Port 1 reset
51:23:35.487368 AHCI#0: Port 0 reset
  • Could the last two lines' strange time-stamp be a clue to what is happening with this guest?
  • Any ideas on how to solve this situation?
Environment:
VirtualBox: Oracle VM VirtualBox VM Selector v6.0.8
Guest and Host: CentOS 7.6
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: A running guest seems to be stuck

Post by socratis »

How do you know that something didn't stop within your guest and VirtualBox is completely irrelevant? Because the lines that you point to aren't crucial errors.

We need to see a complete VBox.log, from a complete VM run, where the problem occurs:
  • Start the VM from cold-boot (not from a paused or saved state) / Observe problem / Shutdown the VM (force close it if you have to).
  • With the VM completely shut down (not paused or saved), right-click on the VM in the VirtualBox Manager and select "Show Log".
  • Save only the first "VBox.log", ZIP it and attach it to your response. See the "Upload attachment" tab below the reply form.
Image
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

Thanks for the advice, socratis.
I have a crontab that restarts any downed guests VMs, so the interesting log may not necessarily be VBox.log, if the heuristic for a new log is creation upon system startup.
In anycase, I've followed your advice, and'll forward the data next time the issue occurs.
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

Today, the guest crashed again (and was revived by my cron script).
Thus, per your suggestion, I'm attaching an archive that contains two log files:
main-direct-ingest-vb-maor-2019-08-23-09-26-56_1.log, which is probably the interesting log, containing the time of the crash.
main-direct-ingest-vb-maor-2019-08-26-09-07-32_0.log, which is the current log, created after my cron revived the guest.
Attachments
virtualbox_crash.tar.gz
logs
(45.32 KiB) Downloaded 15 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: A running guest seems to be stuck

Post by fth0 »

Install the Guest Additions version matching your VirtualBox version.
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

Thanks for your hawk-eyes, fth0.
I'll update the Guest Additions, and see if my issue disappear.
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

After I upgraded the Guest Additions to be in line with the VirtualBox version, the issue happened again,
namely - the guest crashed again (and was revived by my cron script).
Thus, I'm attaching an archive that contains two log files:
main-direct-ingest-vb-maor-2019-08-27-17-29-02_0.log which is the current log, created after my cron revived the guest.
main-direct-ingest-vb-maor-2019-08-27-09-13-33_1.log which is probably the interesting log, containing the time of the crash.

Could you suggest other things that may have caused the crash of the guest?
Attachments
virtualbox_crash_20190827.tar.gz
logs 20190827
(47.79 KiB) Downloaded 12 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: A running guest seems to be stuck

Post by fth0 »

I don't see any sign of a crash in any of the log files. What do you mean by crash? What does your cron script check?
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

The issue happened again today at ~"Fri Aug 30 00:34:19 UTC 2019".
Attached the relevant log files.

The line in my crontab is:

Code: Select all

*/2 * * * * /home/qa/start_vms.sh >/dev/null 2>&1
And /home/qa/start_vms.sh is:

Code: Select all

$ cat /home/qa/start_vms.sh
#!/bin/sh

/usr/bin/VBoxManage startvm "sonarsql-test-virtualbox" --type headless
/usr/bin/VBoxManage startvm "oracle12" --type headless
/usr/bin/VBoxManage startvm "main-direct-ingest-vb-maor" --type headless
/usr/bin/VBoxManage startvm "mariya_pymongo-drivers_vb" --type headless
/usr/bin/VBoxManage startvm "mariya_java-drivers_vb" --type headless
/usr/bin/VBoxManage startvm "mariya_c-drivers_vb" --type headless
So, every two minutes cron checks if a guest VM is not up, and starts it. In our case, the guest is "main-direct-ingest-vb-maor".
Attachments
virtualbox_crash_20190830.tar.gz
logs
(36.45 KiB) Downloaded 13 times
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: A running guest seems to be stuck

Post by fth0 »

Some random observations and ideas:
rbarak wrote:The issue happened again today at ~"Fri Aug 30 00:34:19 UTC 2019".
And the log file started nearly exactly 48 hours before that. There seems to be some regularity here (besides your 2 minute cron interval). Your older log files showed different relative starting times though, maybe indicating something taking place every 6 hours?

The last log entry is at relative time 42:15:49 and complains about the guest being unresponsive.

The VMs are located under a user folder named /local/raid10/qa, especially the virtual disk images and the log files. How reliable Is the availability of this folder?

The cron script uses the VBoxManage startvm command every two minutes without checking the current VM state. According to the VirtualBox User Manual, it should start the VM only when the VM is either in the poweroff state or saved state. When I test it with a running VM, I get an error message. So there seems to be something missing at the end of the log files.

My primary suspect so far is the RAID10 ... searching for RAID in the VirtualBox forums results in several hits ...
rbarak
Posts: 52
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Post by rbarak »

fth0 wrote:The VMs are located under a user folder named /local/raid10/qa, especially the virtual disk images and the log files. How reliable Is the availability of this folder?
/local/raid10/ is mounted on a RAID10 (mirroring with striping) and is very reliable. Short of physical disk error, I've never had a problem with RAID10.
fth0 wrote:The cron script uses the VBoxManage startvm command every two minutes without checking the current VM state. According to the VirtualBox User Manual, it should start the VM only when the VM is either in the poweroff state or saved state. When I test it with a running VM, I get an error message. So there seems to be something missing at the end of the log files.
If I run the VBoxManage startvm while a VM is running, the bellow three error lines are produced, but the machine will continue to run unharmed, if it was already running.

Code: Select all

$ /usr/bin/VBoxManage startvm "main-direct-ingest-vb-maor" --type headless
VBoxManage: error: The machine 'main-direct-ingest-vb-maor' is already locked by a session (or being locked or unlocked)
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
VBoxManage: error: Context: "LaunchVMProcess(a->session, sessionType.raw(), Bstr(strEnv).raw(), progress.asOutParam())" at line 726 of file VBoxManageMisc.cpp
$ vboxmanage list runningvms                                                                                                                                                                  
"main-direct-ingest-vb-maor" {eaf48de3-2eca-47a8-9014-b286c5a5eb99}
$ 
fth0 wrote:My primary suspect so far is the RAID10 ... searching for RAID in the VirtualBox forums results in several hits ...
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: A running guest seems to be stuck

Post by fth0 »

Another idea:

You could enable the VirtualBox Remote Display (RDP) service (which doesn't use the network between host and guest), let the guest run into the problem, and then investigate the guest's state. Use dmesg on the guest, and especially look for the root file system being read-only. ;)
Post Reply