Recovery of VDI snapshots

[drag]

Hello there,

my VM (Linux Ubuntu) is badly damaged after removal of couple of old snapshots. The status changed to "not accessable" so I tried to add a new VM with the base VDI from the old VM. The new VM is working fine but the last 4 months are missing. Is there a way to use the existing snapshots from the last couple of months and recovery the the data out of it?

Thanks a lot for every hint.

BR

[mpack]

Is there a way to use the existing snapshots from the last couple of months and recovery the the data out of it?

Nope, absolutely not. There is no data in them to be recovered. Snapshots are essentially instructions on how to patch an earlier state to get a new state. They are meaningless if the previous state can't be recovered.

So the only data that can be recovered is the stuff still remaining in the chain from base disk to just before the first missing snapshot. Or you can recover everything if you have any actual backups you made. Likewise if you have a backup which is not new but does at least include the deleted snapshots then you can restore the chain and all data can be recovered.

I assume that "removal of some old snapshots" is something you did outside of VirtualBox: inside VirtualBox it would have merged the data before deleting the physical files.


p.s. Experienced users don't use snapshots. They are an accident waiting to happen, and they hog disk space for no reason except to pander to laziness. Make proper offline backups, which have none of these problems. If you want a good fright try Googling for "snapshot problem site:forums.virtualbox.org".

[aeichner]

Hopefully this is of help:

Code: Select all

= Trying to rescue data from broken disk image snapshot chains =

This document describes how to attempt to achieve data recovery in case some of
the differential disk images in a snapshot chain are missing. Depending on the layout
of the snapshot chain and the amount of missing images and where they are located in the
chain it is possible to recover some or under certain circumstances even all data.


== Data recovery process ==

The next section will serve as a guide through the recovery process. It shows how to
create a new disk image chain leaving out all missing images. The following simple
example will be used to illustrate the process:

+ test.vdi {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5} {00000000-0000-0000-0000-000000000000}
    ! test_diff.vdi {f5cf171e-5fa8-46fc-bd38-0dc79f1b6992} {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5}
        + test_diff2.vdi {ed17c0a0-e7de-40da-8179-8e3c96d5aeaa} {f5cf171e-5fa8-46fc-bd38-0dc79f1b6992}

The chain consists of three images where the image in the middle is missing. The first UUID shown
is the image creation UUID which is unique for every image, it serves as a unique identifier for a disk image.
The second UUID is the parent UUID and links any differential image to its logical parent in the image chain.
The first image is called the base image and has a zero UUID because there is no parent. The next
image is a differential image because it contains only differences to the disk image made after the creation of the
image. It also has a unique creation UUID but here the parent UUID is not zero but contains the creation UUID of
the base image to link it to it. The second and last differential disk image test_diff2.vdi in the chain has the parent UUID
set to the creation UUID of the first differential image test_diff.vdi. In the example the second image test_diff.vdi is missing.
To rebuild the chain the parent UUID of the last image must be set to the base image creation UUID so that the link to the first
differential image gets removed.


=== Preparations ===

Prior to any recovery attempt a backup of all existing images involved in the attempt should
be taken to be able to start from anew if a mistake is made.


=== Getting the current parent/child relations in the chain ===

To create the new image chain the current chain with all the missing links should be noted down.
This can be done by either traversing the chain in the virtual media manager [1] and looking at the UUID
entry in the information pane or it can be directly extracted from the VM configuration file by looking
at the media registry which should look like the following for the previous example:

        <HardDisk uuid="{9eee05c8-45a5-4cce-9f63-fdf6d29f10d5}" location="test.vdi" format="VDI" type="Normal">
          <HardDisk uuid="{f5cf171e-5fa8-46fc-bd38-0dc79f1b6992}" location="test_diff.vdi" format="VDI">
            <HardDisk uuid="{ed17c0a0-e7de-40da-8179-8e3c96d5aeaa}" location="test_diff2.vdi" format="VDI"/>
          </HardDisk>
        </HardDisk>

The first image is always the base image and the nested images are the differential ones in the respective order.
The UUID values given there are the creation UUID for the respective medium, the parent UUID can be inferred by looking
at the creation UUID of the medium coming before in the chain.

The end result of the current chain should look like the example at the beginning.

+ test.vdi {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5} {00000000-0000-0000-0000-000000000000}
    ! test_diff.vdi {f5cf171e-5fa8-46fc-bd38-0dc79f1b6992} {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5}
        + test_diff2.vdi {ed17c0a0-e7de-40da-8179-8e3c96d5aeaa} {f5cf171e-5fa8-46fc-bd38-0dc79f1b6992}


=== Setting up the new image chain ===

The next step is to remove the missing images from the chain and change the parent UUID to match the new chain.
In the example the test_diff.vdi image gets removed from the chain and the test_diff2.vdi parent UUID must be changed to
the base image creation UUID. The end result should look like the following:

+ test.vdi {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5} {00000000-0000-0000-0000-000000000000}
    + test_diff2.vdi {ed17c0a0-e7de-40da-8179-8e3c96d5aeaa} {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5}


=== Writing the new parent UUIDs ===

VirtualBox comes with a small disk image utility called vbox-img which is used as a standalone utility to work with
disk images. It doesn't require VBoxSVC running and it should be ensured that VBoxSVC is indeed not running to avoid
any interference with it.

With vbox-img traverse the list of disk images and change the parent UUID to the new one using the following command

{{{
vbox-img setuuid --filename <path to diff image in chain> --parentuuid <new UUID with curly brackets>
}}}

For the example vbox-img needs to be executed only once and it will produce the following output:

{{{
./vbox-img setuuid --filename test_diff2.vdi --parentuuid {9eee05c8-45a5-4cce-9f63-fdf6d29f10d5}
Old image UUID:  ed17c0a0-e7de-40da-8179-8e3c96d5aeaa
Old parent UUID: f5cf171e-5fa8-46fc-bd38-0dc79f1b6992
New parent UUID: 9eee05c8-45a5-4cce-9f63-fdf6d29f10d5
}}}


=== Importing the new disk image chain ===

The new disk image chain needs to be imported into VirtualBox so it can be used with a VM.
The virtual media manager [1] can be used for that by clicking the "Add" toolbar button and
selecting the images in the new chain one by one starting with the base image and moving forward
in the chain. To avoid interference with the original chain containing the missing images the VM
containing the images should be unregistered from VirtualBox. Otherwise VirtualBox will complain
that there are already images with a matching UUID in the media registry.


=== Setting up a VM with the new image chain ===

If the old image chain contained an operating system it is not advised to boot directly from
it because it might try to write to the filesystem causing additional data corruption/loss.
The new chain should be attached to a new VM with a fresh OS install or live system running.
When attaching the repaired chain the last image in the chain must be selected, selecting the base
image will result in creation of a new differencing image and all other data will be inaccessible.

Depending on the amount of data lost the guest operating system of the new VM might not be able to
repair/mount the filesystem from the damaged chain. However not all might be lost in that case,
there are data recovery tools and specialized Linux distributions to help with that but the usage
is highly dependent on the used filesystem and type of data to be recovered so they won't be discussed
here. Use [2] as a starting point for any further recovery attempts. It is advised to create a snapshot
of the VM before any attempt to avoid unwanted writes to the damaged image chain and make it possible to
revert to the unmodified state of the repaired chain.

[1] https://www.virtualbox.org/manual/ch05.html#vdis
[2] https://www.fossmint.com/linux-rescue-recovery-tools/

[drag]

Thanks guys for advices, I certainly didn't carefully study the concept behind snapshots until desaster happened. Likely I could recover some of them using vboxmanage sethduuid and sethdparentuuid and establishing the chain again.
In the next step I used gui and added the snapshots to the base vdi in media manager. After that I changed the storage in VM pointing to the last added snapshot. I was lucky I suppouse since this worked for me. I am still missing the last snapshot and a week of work but this is obviously better then 4 months.

Next time I will make a backup of virtuals VMs in a traditional way.

Thanks again.

[mpack]

Likely I could recover some of them using vboxmanage sethduuid and sethdparentuuid and establishing the chain again.

I'm not as optimistic as Aeichner about that. The procedure he outlines hacks the chain so that VirtualBox doesn't know that pieces are missing. It does not recover the data that was lost, and if the deleted files were large (why else delete them) then data loss is going to be substantial, and very hard to determine.

A snapshot is not any kind of backup (*), so it isn't a matter of traditional backups vs snapshots. You either have a "traditional" backup or don't have a backup at all.

(*) Snapshots encode a choice of guest states, and only do that when 100% intact. They do nothing at all for catastrophes perpetrated on the host.

[aeichner]

It does not recover the data that was lost, and if the deleted files were large (why else delete them) then data loss is going to be substantial, and very hard to determine.

This depends a lot on the depth of the snapshot chain and where in the chain the images are missing and what kind of files you are primarily after in the VM. Sure, if the latest differential image is already missing there is no chance to recover that data. However if one of the earliest snapshots is missing and the same file is updated in the snapshots coming after that again chances are high you only lost the earliest state but not the later states which are most of the time the ones people are after in these scenarios.

The differential image captures all writes since the point where it was created. So if you change a file in the VM, create a snapshot and then change and save the file again you can recover these changes and as VDI has a 1MiB block size it will always read the rest of the data from the underyling image in case the write is not aligned to the block size so you end up
with quite a lot of possibly not changed data in the latest snapshot as well. Huge video files can't be recovered of course (but who does video processing in VirtualBox anyway) but most of the time people are after rather small office documents and source code projects where the chances are high you can recover a lot given you have the latest differential images in the chain. WIth VMDK where the block size is only 64KiB or QCOW1 which only has 4KiB you might not be so lucky of course. Btw. these procedures have already helped others in the past to recover months of work on projects.

You are absolutely right with the statement about backups though. Snapshots are in NO way Backups!

[mpack]

This depends a lot on the depth of the snapshot chain and where in the chain the images are missing and what kind of files you are primarily after in the VM.

Actually no, the data loss exists regardless, unless the snapshot was empty. Whether the data loss is important depends on a great many things including some things you mention, and others which are more subjective (i.e. I don't know what importance the OP attaches to his files).

Anyway I think you are guilty of assuming that most files are less than 1MB. This may be true for a software developer, but IME not true in general - it's not just video files that exceed 1MB. Likewise, I am probably guilty of assuming the opposite. Obviously there is no universally correct assessment of importance - only time will tell what the effect on the OP is.