virtualbox.org

Hello Guys

Yes I admit it: I'm writing this post only because I'm frustrated. Very, very frustrated. With VirtualBox. The kind where you'd like to grab your computer and throw it through the closed window out into the streets. And where you then go down the stairs, wish you'd live in the 10th floor instead of the first because the hit would have been harder, grab your pc, go back up and throw it out of the window again. Another one that is still intact because the shattering glass gives you some relief. At least a little.

I was working in my Kubuntu install inside VirtualBox under Windows. At 3 AM in the morning. Full-screen mode. Bad idea. The Windows-message about the imminent shutdown for an Update did not make its way into that mode. It took me by surprise. Not a nice moment. But everything not too bad. You'd think. I'm a notorious Ctrl+S hitter. You know - Where all your stuff is saved to hard disk? I do it by habit. Everywhere. All the time. To the point where it can get annoying to people who watch me work.
So I wait for Windows to complete its destructive work. And I know I could have disabled Auto-Updates. I didn't. The computer starts back up. I start up VirtualBox to have a look at how much I've lost. I'm expecting minutes. I get days. And not thanks to VirtualBox, but thanks to Github. It tells me my VirtualBox is in an unrecoverable state. The *.vbox-file is gone. I mean hello?! Gone!!! You'd think VirtualBox loads the information it needs from the file and would write information back only if the user changes the configuration. But how on earth would the file get deleted?! Well okay - It's basically just an XML-file and I can either copy one from another vbox and make the necessary adjustments, like add the path to the hard disk file or I can create a completely new machine configuration and mount the disk on that one. So far so good. Annoying - yes. But acceptable. I restart the new configuration from my hard disk install. Kubuntu starts up. First surprise: My background and shortcut settings are gone. Huh - How is that? These surely have been saved long ago.
Oh-oh, my files!
I start up the console, check my folders. Can't believe my eyes. Files missing. Files in an old state. Stuff gone I have saved to disk a hundred times. It must be a state from like a month ago. Genuinely weird. Github is god that moment. I test a few of the programs I installed and realize some of them don't work anymore. Crashes without error. Broken and no apparent way to fix it. Looks like I'll have to do a fresh install from scratch. Two days coding lost for good. *Part deleted when cooled off the next day*

Sounds like you used Snapshots in the machine but did not include them in the new VM config you've put together.
The good way would have been to copy the .vbox-prev to .vbox.
If you did indeed have snapshot or saved state, and you modified the base disk meanwhile, then it's over for the snapshots...

It's far more likely that, rather than Virtualbox "sucking", Franz, you unfortunately "thrashed about" (saw that verbage in a different post, I think it's Mpack's expression, regarding trying anything that pops to mind to fix a problem, rather than asking questions beforehand how to solve it.)

Questions about how to handle Auto Updates for Windows (disable auto-restart & use a script, etc.), missing .vbox files (open the guest folder & rename .vbox-prev, etc.), recovering unrecoverable guests, etc. have been discussed multiple times on the forum, and can be discussed again to help your situation out.

If anything can be done, see this: viewtopic.php?f=24&t=48476 to provide what things might be needed.

As for backups, Virtualbox doesn't specifically have a backup routine for guests - Virtualbox provides virtual hardware, like a new PC provides physical hardware. You have to install backup software on a PC to back it up live, or shut down the PC and image the hard drive from a boot CD to back it up shut-down. The PC itself won't do that automatically unless you provide a way for it to do so. Same as a Virtualbox guest. Backup software installed within the guest, or copy the guest folder & hard drive file while the guest is shut down. (Frankly, these considerations should have been considered before one starts doing important things on any PC, physical or virtual.)

Seems to me this is likely a pebcak error, quite unfortunately. Pebcaks tend to cause cluster-cusses of epic proportions when thrashing occurs thereafter. Try to post what the linked forum page asks for and we'll see what can be done to recover.

noteirak wrote:Sounds like you used Snapshots in the machine but did not include them in the new VM config you've put together.

Thank you, this is true.

So to recap, what happened? Where was the newer data stored and at which point and how was it lost when I made my "half knowledgable" recovery attempt? Or more specific: How did the snapshot mess with my assumption that starting from my disk - just with real hardware - would give me the system back at its latest state and only stuff in RAM would be lost?

To clarify: I set the {xxx-xxxx-xxx-xxxx}.vmdk as disk for the new configuration, not the original *.vmdk-file.

For each snapshot taken, a new diff image will be created (the {xxx-xxxx-xxx-xxxx}.vmdk file) next to the original disk file.
I don't know how many snapshots you had, but from what you are saying, you basically re-created the VM with the wrong file, either the base disk or a diff file which was not the latest.

The problem in doing that is that the file itself is not flagged as immutable, so it's very much possible to make VirtualBox use the base disk, or a snapshot file in between and run on it.
If you do that, the "next" snapshot file will be out of sync because it relies on the previous snapshot file to not have changed since the snapshot was taken. There is no going back from that. There is no way to know what changed then.

From what you say in your original post, it all points to the wrong file being choosen, regardless of what you say now.

I had only one snapshot and used its {xxx-xxxx-xxx} file as disk for the new state. So this one should have been the latest if I understand you correctly. However as base config file I used one from another machine and only changed the disk in my text editor. Did that somehow destroy the back link from the snapshot disk to the base disk? Thanks.

Btw. Sorry for the swearing in my first post.

if it was missing the base disk, it wouldn't even start at all. I am not sure about how the whole thing works out with VMDK tbh.

In VirtualBox, a VMDK difference chain works the same as a VDI chain. Same UUID linkage, same control info in vbox file. Same potential for disaster. Only the minor detail of file format is different.

Well what I did was changing the disk path in the vbox file to the path of the {xxx-xxx}-file. Nothing else. It then did boot without errors or anything but the data situation was the one encountered in the first post. It also was not the state of the snapshot since e.g. a few installations that worked at the time of the snapshot (I did it after successfully installing all libraries, programs and tools i needed) were there but couldn't be started.

This is weird because if I understand you correctly it should have either not been bootable at all or I should have encountered either the state I was expecting since using the correct snapshot branch or then the base disk state. But it is/was neither of the two.

Booting without errors doesn't really tell you anything. A snapshot chain represents multiple points in time (states). Booting without errors in an offshoot VM tells you that one of those states can be recovered - usually the oldest one. The act of doing so will often corrupt the dependent states. Since each state is stored as the difference from the next older states, modifying the oldest state corrupts the entire chain.

Yes I get that. But if you've made only one snapshot and you're booting from the one {xxx-xxx-xxx.vmdk}-file there is, and you additionally check that this had been written on the same day, and you haven't worked in any other VM, by your logic: Shouldn't that boot the newest and not "some random point somewhere in the chain"? Were other possibly invalid statements from my new *.vbox-file able to corrupt that state?

To recap what happened, from the beginning to the end (no work was done on any other VM during this timespan):
- Setup Kubuntu 14.04 in a new VM with a new vmdk-disk
- Get programs, libraries and code I expected to need more often
- Make one snapshot after checking that everything runs as expected (1)
- Work, create new stuff
- Safe stuff to disk (2)
- Some more work
- Host shutdown for update
- Restart: *.vbox-file is missing
- Instead of copying the *.vbox-prev -file to a *.vbox-file as I should have I cloned the *.vbox-file of another, old project (4) that had nothing to do with the current project (different hard disk file too) but was created on the same machine. I renamed it in the xml, set the machine uuid to what VirtualBox expected for the project and altered the <HardDisk>-tag content to fit the {xxx-xxx-xxxx.vmdk} file I expected to hold the content from (2) since it was the only *.vmdk-file that was altered the same day (3)
- Configuration (3) booted successfully but did not hold the expected data. Not only files and modifications were missing, also some installations seem corrupted.
- I posted here on the forum
- Possibly unfortunately I also did a boot from the original *.vbox-file (1) right thereafter, which holds the old data and working installations as expected. I didn't change anything manually in that setup but I presume starting it up already will invalidate the later snapsho at that point if it might have been still recoverable at that point?

I discovered that the new, not working setup has not been written to the {xxxx-xxxx} file, but to a new *.vdi-file, I assume because it didn't know about the snapshot and therefore created a new machine in the regular disk path. This did not overwrite my "basic" disk state.

Here is what still exists:
- "{xxx-xxx-xxx}.vmdk" - file with a last changed time stamp that approximately fits the time of the windows host shutdown.
- "xxx-xxx.sav" - file, time stamp indicates to me that it could have been approximately created when I did the snapshot
- Original "Kubuntu2015.vmdk" - file but the time stamp tells me it changed an hour after the {xxx-xxx}-file which would fit the time when I did a boot from it as written above.


Any chance that this is still recoverable and the not working {xx-xxxx}-boot (3) was due to an incorrect *.vbox-file that possibly used the other machine's base disk (4) as a base for the path (Can that happen?) ? (4) was an install of Kubuntu 14.04 as well with similar installs on it.

Thank you for your time!

Okay, so looking through the files I still had as listed above it got me hoping that I might be able to actually fix it, since the snapshot file apparently even was able to boot with no base disk, the base disk was still there and not really altered at all and my changes on the snapshot had been saved to a new disk (lucky me, thanks VirtualBox!). Since no *.vbox or *.vbox-pref file was left of the original setup I chose to yet again base my implementation on another vbox. But now I used an old vbox in which I especially created a snapshot to see how the markup had to look like. That was enough to figure out the right markup to write a new *.vbox-file. Started up and it is all back! Boy I got lucky there, PEBCAK not that disastrous for once! Thank you all for your inputs, they sure got me on the right track!

So now only out of interest: What happened in the previous failed attempt when I did load the snapshot disk file without any linkage to the base in the *.vbox-file? I'm really surprised it was able to start up at all.

There are several possibilities. VirtualBox will be able to find any UUID-linked parent media if that media is registered in any registered VM. If you have a habit of copying VDI files around then its possible that it was still registered in the original VM.

Another possiblity is that it wasn't really a snapshot. Sometimes users make clones of snapshot files, with the clone still having a name that makes it look like a snapshot file, without realizing that a clone is stand alone. VirtualBox doesn't care if you replace a difference image with a clone, provided the filename and UUID are correct.

There are probably lots of other explanations.

Another possiblity was that the snapshot existed for enough time that all necessary sectors for the OS and your main programs were at least written to since the snapshot was taken.
I would be interested in the sizes of the main disk and the snapshot