VM Stuck in "offline snapshotting" state (#19074)

Discussion about using the VirtualBox API, Tutorials, Samples.
Post Reply
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

VM Stuck in "offline snapshotting" state (#19074)

Post by Brian of the Pines »


ModEdit; related ticket: #19074: VM Stuck in "offline snapshotting" state
Hi. I'm working on a Perl+Curses TUI front-end for a headless VirtualBox server. It uses the vboxwebsrv webservice via the SOAP::Lite package to interact with VirtualBox. I am following the API documented in the latest SDK from Oracle (6.0.10).

When my scripts initiate a snapshot of a powered-off machine via IMachine::takeSnapshot, the VM then becomes stuck in "offline snapshotting" state ("stuck" as in it's been in that state for over 12 hours). As far as I can tell, no useful messages are written to either the VM's log (VBox.log) or the user's VBoxSVC.log. This is what the script does:
  1. Get an ISession object via IWebSessionManager::getSessionObject
  2. Lock the session in Shared mode via IMachine::lockMachine
  3. Get a mutable IMachine object with ISession::getMachine
  4. Initiate the snapshot from the mutable IMachine object, getting an IProgress object, with IMachine::takeSnapshot
  5. Start a modal window which periodically queries the IProgress interface for its status until IProgress::complete returns true
As I'm reviewing my code to write this, I note that it's possible that ISession::unlockMachine is being called before the IProgress object reports success. Although the documentation implies that a session remains locked as long as it is busy even after a call to ISession::unlockMachine, I wonder if this may be causing VirtualBox to do something bad resulting in the stuck "offline snapshotting" state.

Primary question:
How do I safely get the VM out of this "offline snapshotting" state? I've run into this more than once as I'm working on this set of scripts, and have rendered one VM completely unbootable in my attempts to regain control of it. I'm loath to corrupt another one.

Aside: Is there a better place to pose questions involving the webservice? Would this be more appropriate in the "Third Party Applications" thread?

Environment info:
  • Host OS is Debian Buster 64-bit.
  • Guest OS is Windows 7 Ultimate 32-bit.
  • VBoxHeadless --version gives 6.0.10r132072.
  • VBox.log does not get anything written to it.
Last edited by socratis on 10. Nov 2019, 10:41, edited 1 time in total.
Reason: Added ticket related information.
I'm not really an idiot; I just play one on TV.
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

Re: VM Stuck in "offline snapshotting" state

Post by Brian of the Pines »

Two months later, and nobody has any advice? :-(
I'm not really an idiot; I just play one on TV.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VM Stuck in "offline snapshotting" state

Post by mpack »

Without a log, what would any advice be based on? Yes, I saw where you said that no log is written (which I don't really believe btw - if the VM starts, it created a log), but that doesn't change the reality in any way.

Plus you're using the software in a really obscure way, so the chances that someone else has seen the same problem seem low. If you can't do remote snapshots reliably then I'd say - stop doing remote snapshots, and maybe raise a BugTracker ticket about it.
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

Re: VM Stuck in "offline snapshotting" state

Post by Brian of the Pines »

First, thank you for responding.
mpack wrote:maybe raise a BugTracker ticket about it.
That is reasonable advice.
mpack wrote:you said that no log is written (which I don't really believe btw
Please don't accuse me of lying. If I said no log file is written, it's because no log file was written. Observe:

Code: Select all

$ vboxmanage showvminfo "Windows 10 Build Environment" | grep "tate"
State:                       offline snapshotting (since 2019-09-05T19:25:17.936000000)
$ pwd
/var/VirtualBox/VirtualBox VMs/Windows 10 Build Environment/Logs
$ ls -lh VBox.log
-rw------- 1 virtualbox virtualbox 188K Sep  5 17:45 VBox.log
In particular, look at the times. The machine shifted into offline snapshotting state at 19:25, but the log was written at 17:45, almost two hours earlier.
If that alone doesn't convince you, here's the last line of that particular log file:

Code: Select all

00:05:24.485189 Console: Machine state changed to 'PoweredOff'
That log does not correspond to the currently-stuck-snapshotting state of the VM.
Also:
mpack wrote:if the VM starts, it created a log
Yes, but this is an offline snapshot. The VM wasn't running when the snapshot operation was started. Ergo, no log.
mpack wrote:then I'd say - stop doing remote snapshots
Thanks for that. But the VM is already stuck in this "offline - snapshotting" state (*points at post title*). I can't do anything to it while it's in that state, and apparently nobody knows how to get it out of that state (*points at the boldface question in the original post*).

Rebuttals aside, taking my issue to BugTracker is probably the next move to make.
I'm not really an idiot; I just play one on TV.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: VM Stuck in "offline snapshotting" state

Post by socratis »

Brian of the Pines wrote:
  1. Get an ISession object via IWebSessionManager::getSessionObject
  2. Lock the session in Shared mode via IMachine::lockMachine
  3. Get a mutable IMachine object with ISession::getMachine
  4. Initiate the snapshot from the mutable IMachine object, getting an IProgress object, with IMachine::takeSnapshot
  5. Start a modal window which periodically queries the IProgress interface for its status until IProgress::complete returns true
This needs to move to the "VirtualBox API" section from the "Linux Hosts" one. And since it's moving there, read first the "API Discussion Forum - Posting Guidelines", especially the red parts.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

Re: VM Stuck in "offline snapshotting" state

Post by Brian of the Pines »

Thank you, socratis. It looks like you moved the thread already, so I don't need to do anything on that aspect.

I will get to work on a MCRE. Given the dependence on SOAP::Lite, I don't know how "minimal" it'll end up being, but we'll see. I should at least be able to separate it from curses. And maybe I'll discover something in the process.
I'm not really an idiot; I just play one on TV.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: VM Stuck in "offline snapshotting" state

Post by socratis »

Brian of the Pines wrote: It looks like you moved the thread already, so I don't need to do anything on that aspect.
You couldn't do anything on that aspect, (thankfully) only mods can... ;)

Keep in mind that the amount of developers around this site is pretty minimal, I'd say less than 1%. That's the reason you didn't originally receive any answers. I, for example, won't be able to help you any further I'm afraid, not a programmer... :?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VM Stuck in "offline snapshotting" state

Post by mpack »

Brian of the Pines wrote: Please don't accuse me of lying.
You're the only one here making accusations - I said I believed that the information is false, I made no suggestion that the falsehood was intentional. But, if that's your threshold for taking offence then it's probably best if I don't respond to you at all.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: VM Stuck in "offline snapshotting" state (#19074)

Post by socratis »

Brian of the Pines wrote:and apparently nobody knows how to get it out of that state
Have you tried killing the VBoxSVC service? Does that "ruin your day"?

Tied the ticket and the thread...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

Re: VM Stuck in "offline snapshotting" state

Post by Brian of the Pines »

socratis wrote:Have you tried killing the VBoxSVC service? Does that "ruin your day"?
I've tried that a while back, and I seem to recall it did bad things (although at the moment I don't remember exactly what bad things happened :oops: ). I've also tried rebooting the host machine, but that doesn't have any apparent effect.

I need to spin up a fresh one and export it immediately; that way I can at least blow away the stuck one and not have to spend all day getting a new VM up in its place. Then at least testing this situation won't be quite as painful..
I'm not really an idiot; I just play one on TV.
Brian of the Pines
Posts: 9
Joined: 30. Aug 2019, 18:07
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Windows 7 Ultimate, Windows 10

Re: VM Stuck in "offline snapshotting" state (#19074)

Post by Brian of the Pines »

An update summarized from the ticket, if anyone else is interested:

The cause of the machine getting stuck in "offline snapshotting" is definitely tied to unlocking the session before the operation is completed.

I discovered that my script is doing this because, in part, IProgress::Completed always immediately returns true, whether or not the process is actually complete. Modifying my script to monitor IProgress::Percent instead (wait for it to reach 100) eliminated the problem.

Getting the machine out of the stuck state is a gamble and seems unpredictable. Stopping the VBoxSVC may allow the machine to return to "powered off", but in some cases results in the VM being left in an inaccessible state, due to errors such as the following:
VBoxManage: error: Hard disk '/var/VirtualBox/VirtualBox VMs/Windows 10/Windows_10-disk001.vmdk' with UUID 
   {4687eab9-1a50-4612-afe1-bcfa39e35fc8} cannot be directly attached to the virtual machine 'Windows 10' 
   ('/var/VirtualBox/VirtualBox VMs/Windows 10/Windows 10.vbox') because it has 1 differencing child hard disks
   VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine
Irritatingly, I haven't been able to get back to that particular kind of failure in order to further investigate whether deleting the snapshot vmdk and removing it from the vbox XML will restore access to the machine.

In any case, once VBoxSVC is stopped, I can at least use "vboxmanage unregistervm", blow the whole thing away, and restore it from an export (assuming I made one).

I'll come back with another update if I learn anything else about it. I'm hoping the kind dev that's engaged with me on the BugTrack ticket will see this through to a fix, rather than just a workaround.
I'm not really an idiot; I just play one on TV.
Post Reply