Mysterious problem: Changes of the last 3 days omitted

[tY Software]

Hi,

This is the weirdest problem I've ever seen..

I've got a Debian server with a wiki, one issue tracker, subversion and WebSVN running in a vm on my Win7 host.
To backup this data i developed an (c#) application which is scheduled to run once a week and does the following (with VBoxManage):

- Get information about vm
- Shutdown per ACPI
- Copy VDI to temporary location
- Start vm
- Package copy of VDI, add vm info to the package and move to target location

This application works perfectly and I never had any problems. It generates a rich log output, so i can always exactly see what was going on.

Today (backup was running at 4:00 am) i realized that the server did not restart because the VDI was not attached to it any more. I've got no idea how this could happen. My application isn't doing this and beside it no other software interacts with it.
When i re-attached the VDI to the vm and restarted I quickly realized that all changes made the last 3 days were lost.
I also checked the backup and it was in the same state.

How can this be? The log of my application didn't show any probkems. There are no differences to the last times.

I really have no idea what could have caused this strange problem. How can data of exactly last 3 days (and not more) be lost? There are no other backups running (which could have been restored in any mysterious way).

Has anybody some kind of idea where this incredible behavior could come from? I'm using the latest version of VBox.

Thanks in advance!

[mpack]

You don't mention snapshots. Were you using any?

If not then there isn't any way I can think of for VirtualBox to cause this fault. Apart from snapshots (and related features such as immutable drives and linked clones), VBox has no memory of past states of VDI files - so even if it was buggy as hell it simply doesn't have the data to revert to a past state. If non of these VBox features was involved, and the VM hasn't simply been non-functional for 3 days, then I'd suspect that someone has done a system restore on your host.

[tY Software]

Hi,

Thanks for the quick reply! No, there were no snapshots. But this is a good point. I took a snapshot some days ago (don't rember exact time). When I wanted to delete the snapshot first, there was an error which didn't let me delete it immediately. Then I tried to take another one to see if it works, but this also didn't. Few minutes later I tried it again and it worked.
I think this could be an interesting point.

How should anybody have restored the vm? There wasn't any copy, snapshot, backup or anything else of this day. Even if someone had wanted to restore this point (I don't know who, not many people except me have access), he just could have restored the last backup (1 week ago).

[mpack]

How should anybody have restored the vm? There wasn't any copy, snapshot, backup or anything else of this day. Even if someone had wanted to restore this point (I don't know who, not many people except me have access), he just could have restored the last backup (1 week ago).

Well, that includes information not previously available to me, so yes, the additional info would seem to exclude one of the possibilities I mentioned.

On snapshots, well obviously the ability to "revert to an old state" is one of the fundamental feature of snapshots. It's possible to do this accidentally if you manipulate the vbox files outside of VBox without really understanding the consequences. One scenario might be, e.g. if, lets say, three days after a snapshot was taken you had somehow restored an old .vbox file dating from before the snapshot then essentially the new snapshot would be forgotten and you would revert to the three day old state.

You should be able to look into the snapshot folder and see if a more recent difference file is still present (one with a modification date that comes after the 3 days). If it is then you didn't use the GUI to revert. You could recover the lost data by cloning this recent file with CloneVDI, then building a new VM around the clone VDI - and I'd suggest that in future you avoid snapshots like the deathtrap they are.

[tY Software]

Hi,

thanks for your help! Like mentioned above I deleted this snapshot immediately after I didn't need it any more. I just use snapshots when I apply some big changes to the server configuration. When everything works like expected this snapshot is deleted, otherwise restored. When a backup is running, there are no snapshots.

But I realized that there was a file left in the snapshots folder. This snapshot was not displayed in the VBox gui and was this one I couldn't delete at first attempt. Maybe this snapshot is the root of this problem. Now I manually deleted the file and hope this solves the problem. (It also seems like this snapshot contained the lost information).

It seems like following happened:
When I wanted to delete the snapshot the error was more serious than I've thought. The second attempt removed the snapshot from vm config, but didn't merge or delete the file. After I deleted the snapshot I didn't make any changes, so I realized that data was lost after backup failed to restart vm. But I still don't know why the VDI was not attached to the vm any more.

I know the problems with snapshots and backups, so my backup application always checks and throws an error if there are any.

Edit: I never edit the VBox files in my own and like already mentioned I never keep snapshots to which I could accidentally revert. If I would, the backup would not work like expected.

[mpack]

Now I manually deleted the file and hope this solves the problem. (It also seems like this snapshot contained the lost information).

I don't see how manually deleting a file which isn't being used by the VM will do anything except recover the host disk space. You will still be missing three days of data. However, manually deleting that difference file without keeping backup does preclude the possibility of recovering the data it contained using CloneVDI as I previously suggested - but if you're happy anyway then I guess we're done here.

Edit: I never edit the VBox files in my own and like already mentioned I never keep snapshots to which I could accidentally revert. If I would, the backup would not work like expected.

There is more than one way that the vbox files can be manipulated, you don't need to physically edit one with a text editor. E.g. renaming a "vbox-prev" or "vbox-tmp" files - or otherwise restoring a backup copy of the vbox file - also constitutes changing the contents of the vbox file, and I rather suspect that may have happened here. Snapshot merge failures often leave you with an inaccessible VM. If you read some of the forums here you might have renamed the vbox-prev file, which does indeed recover the VM, however it causes the VM to lose sync of any snapshot changes made in the inaccessible session - i.e. you could be left with an orphaned snapshot and three days of data missing.

[tY Software]

Again thanks for your help.

Fortunately I was the only one who had commited data to the server these three days. When I realized that the missing data is contained in a snapshot file which shouldn't have existed any more, I already re-committed the work (and more) from my working copy. So the only thing lost are few revisions (not the data itself but just the structure) and a tool I quickly re-installed.
So this is not a problem. I just hope this error doesn't occur again.

The only possible reason for this error i see is this half-attached snapshot. So I think now the snapshot is deleted there will be no more chance this will also happen the next time a backup is running.