A running guest seems to be stuck

Discussions related to using VirtualBox on Linux hosts.

A running guest seems to be stuck

Postby rbarak » 16. Aug 2019, 19:46

I have a Guest that is not answering (even to ping)
Code: Select all   Expand viewCollapse view
$ ping -c1 main-direct-ingest-vb-maor.local
PING main-direct-ingest.local (192.168.1.76) 56(84) bytes of data.
From virtualbox01.local (192.168.1.140) icmp_seq=1 Destination Host Unreachable

--- main-direct-ingest.local ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

but still shows in the running list
Code: Select all   Expand viewCollapse view
$ vboxmanage list runningvms  | grep main-direct-ingest
"main-direct-ingest-vb-maor" {eaf48de3-2eca-47a8-9014-b286c5a5eb99}

The last lines in the VBox.log are:
Code: Select all   Expand viewCollapse view
00:01:17.940845 VMMDev: Guest Log: 00:00:00.000373 main     Process ID: 5405
00:01:17.940848 VMMDev: Guest Log: 00:00:00.000373 main     Package type: LINUX_64BITS_GENERIC
00:01:17.942398 VMMDev: Guest Log: 00:00:00.001923 main     5.2.20 r125813 started. Verbose level = 0
00:01:24.612349 Display::i_handleDisplayResize: uScreenId=0 pvVRAM=00007f1697000000 w=1024 h=768 bpp=32 cbLine=0x1000 flags=0x0 origin=0,0
51:23:16.479747 AHCI#0: Port 1 reset
51:23:35.487368 AHCI#0: Port 0 reset

  • Could the last two lines' strange time-stamp be a clue to what is happening with this guest?
  • Any ideas on how to solve this situation?

Environment:
VirtualBox: Oracle VM VirtualBox VM Selector v6.0.8
Guest and Host: CentOS 7.6
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby socratis » 16. Aug 2019, 23:40

How do you know that something didn't stop within your guest and VirtualBox is completely irrelevant? Because the lines that you point to aren't crucial errors.

We need to see a complete VBox.log, from a complete VM run, where the problem occurs:
  • Start the VM from cold-boot (not from a paused or saved state) / Observe problem / Shutdown the VM (force close it if you have to).
  • With the VM completely shut down (not paused or saved), right-click on the VM in the VirtualBox Manager and select "Show Log".
  • Save only the first "VBox.log", ZIP it and attach it to your response. See the "Upload attachment" tab below the reply form.
Image
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
socratis
Site Moderator
 
Posts: 25285
Joined: 22. Oct 2010, 11:03
Location: Greece
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5

Re: A running guest seems to be stuck

Postby rbarak » 23. Aug 2019, 19:41

Thanks for the advice, socratis.
I have a crontab that restarts any downed guests VMs, so the interesting log may not necessarily be VBox.log, if the heuristic for a new log is creation upon system startup.
In anycase, I've followed your advice, and'll forward the data next time the issue occurs.
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby rbarak » 26. Aug 2019, 19:11

Today, the guest crashed again (and was revived by my cron script).
Thus, per your suggestion, I'm attaching an archive that contains two log files:
main-direct-ingest-vb-maor-2019-08-23-09-26-56_1.log, which is probably the interesting log, containing the time of the crash.
main-direct-ingest-vb-maor-2019-08-26-09-07-32_0.log, which is the current log, created after my cron revived the guest.
Attachments
virtualbox_crash.tar.gz
logs
(45.32 KiB) Downloaded 4 times
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby fth0 » 27. Aug 2019, 00:01

Install the Guest Additions version matching your VirtualBox version.
fth0
 
Posts: 148
Joined: 14. Feb 2019, 03:06
Location: Germany
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, macOS, ...

Re: A running guest seems to be stuck

Postby rbarak » 27. Aug 2019, 02:04

Thanks for your hawk-eyes, fth0.
I'll update the Guest Additions, and see if my issue disappear.
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby rbarak » 28. Aug 2019, 02:49

After I upgraded the Guest Additions to be in line with the VirtualBox version, the issue happened again,
namely - the guest crashed again (and was revived by my cron script).
Thus, I'm attaching an archive that contains two log files:
main-direct-ingest-vb-maor-2019-08-27-17-29-02_0.log which is the current log, created after my cron revived the guest.
main-direct-ingest-vb-maor-2019-08-27-09-13-33_1.log which is probably the interesting log, containing the time of the crash.

Could you suggest other things that may have caused the crash of the guest?
Attachments
virtualbox_crash_20190827.tar.gz
logs 20190827
(47.79 KiB) Downloaded 3 times
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby fth0 » 29. Aug 2019, 00:49

I don't see any sign of a crash in any of the log files. What do you mean by crash? What does your cron script check?
fth0
 
Posts: 148
Joined: 14. Feb 2019, 03:06
Location: Germany
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, macOS, ...

Re: A running guest seems to be stuck

Postby rbarak » 30. Aug 2019, 19:01

The issue happened again today at ~"Fri Aug 30 00:34:19 UTC 2019".
Attached the relevant log files.

The line in my crontab is:
Code: Select all   Expand viewCollapse view
*/2 * * * * /home/qa/start_vms.sh >/dev/null 2>&1


And /home/qa/start_vms.sh is:
Code: Select all   Expand viewCollapse view
$ cat /home/qa/start_vms.sh
#!/bin/sh

/usr/bin/VBoxManage startvm "sonarsql-test-virtualbox" --type headless
/usr/bin/VBoxManage startvm "oracle12" --type headless
/usr/bin/VBoxManage startvm "main-direct-ingest-vb-maor" --type headless
/usr/bin/VBoxManage startvm "mariya_pymongo-drivers_vb" --type headless
/usr/bin/VBoxManage startvm "mariya_java-drivers_vb" --type headless
/usr/bin/VBoxManage startvm "mariya_c-drivers_vb" --type headless


So, every two minutes cron checks if a guest VM is not up, and starts it. In our case, the guest is "main-direct-ingest-vb-maor".
Attachments
virtualbox_crash_20190830.tar.gz
logs
(36.45 KiB) Downloaded 3 times
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby fth0 » 30. Aug 2019, 23:32

Some random observations and ideas:

rbarak wrote:The issue happened again today at ~"Fri Aug 30 00:34:19 UTC 2019".

And the log file started nearly exactly 48 hours before that. There seems to be some regularity here (besides your 2 minute cron interval). Your older log files showed different relative starting times though, maybe indicating something taking place every 6 hours?

The last log entry is at relative time 42:15:49 and complains about the guest being unresponsive.

The VMs are located under a user folder named /local/raid10/qa, especially the virtual disk images and the log files. How reliable Is the availability of this folder?

The cron script uses the VBoxManage startvm command every two minutes without checking the current VM state. According to the VirtualBox User Manual, it should start the VM only when the VM is either in the poweroff state or saved state. When I test it with a running VM, I get an error message. So there seems to be something missing at the end of the log files.

My primary suspect so far is the RAID10 ... searching for RAID in the VirtualBox forums results in several hits ...
fth0
 
Posts: 148
Joined: 14. Feb 2019, 03:06
Location: Germany
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, macOS, ...

Re: A running guest seems to be stuck

Postby rbarak » 14. Sep 2019, 00:45

fth0 wrote:The VMs are located under a user folder named /local/raid10/qa, especially the virtual disk images and the log files. How reliable Is the availability of this folder?

/local/raid10/ is mounted on a RAID10 (mirroring with striping) and is very reliable. Short of physical disk error, I've never had a problem with RAID10.

fth0 wrote:The cron script uses the VBoxManage startvm command every two minutes without checking the current VM state. According to the VirtualBox User Manual, it should start the VM only when the VM is either in the poweroff state or saved state. When I test it with a running VM, I get an error message. So there seems to be something missing at the end of the log files.

If I run the VBoxManage startvm while a VM is running, the bellow three error lines are produced, but the machine will continue to run unharmed, if it was already running.
Code: Select all   Expand viewCollapse view
$ /usr/bin/VBoxManage startvm "main-direct-ingest-vb-maor" --type headless
VBoxManage: error: The machine 'main-direct-ingest-vb-maor' is already locked by a session (or being locked or unlocked)
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
VBoxManage: error: Context: "LaunchVMProcess(a->session, sessionType.raw(), Bstr(strEnv).raw(), progress.asOutParam())" at line 726 of file VBoxManageMisc.cpp
$ vboxmanage list runningvms                                                                                                                                                                 
"main-direct-ingest-vb-maor" {eaf48de3-2eca-47a8-9014-b286c5a5eb99}
$


fth0 wrote:My primary suspect so far is the RAID10 ... searching for RAID in the VirtualBox forums results in several hits ...
rbarak
 
Posts: 31
Joined: 25. Sep 2018, 20:43

Re: A running guest seems to be stuck

Postby fth0 » 15. Sep 2019, 10:32

Another idea:

You could enable the VirtualBox Remote Display (RDP) service (which doesn't use the network between host and guest), let the guest run into the problem, and then investigate the guest's state. Use dmesg on the guest, and especially look for the root file system being read-only. ;)
fth0
 
Posts: 148
Joined: 14. Feb 2019, 03:06
Location: Germany
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, macOS, ...


Return to VirtualBox on Linux Hosts

Who is online

Users browsing this forum: mrvanes, socratis, tarpenbek and 24 guests