Page 1 of 2

VDI / VHD Repair Tool

Posted: 28. Sep 2009, 07:02
by ProVega
I would love to see an "internal command" or other feature that would "repair" common problems with corrupted VHD/VDI files. Perhaps rebuild the header? Perhaps validate checksums / lengths / other common problems? At least something to make the drive mountable by Windows or something to get files off of it.

Ideally this tool would be able to bring a VDI back to life after a BSOD on the host or other issue caused by Virtual Box crashing on the host.

Re: VDI / VHD Repair Tool

Posted: 28. Sep 2009, 11:13
by mpack
Checksumming and other validation checks only apply to headers and are already applied every time the VM starts. There isn't much that VBox can do to protect guests data: that sort of thing is up to the guest. Dealing with "a power cut" is a common scenario that any guest should be able to recover from.

Anyway, back to the headers: checksums and so forth allow you to detect the corruption, not correct it. Correction would require redundancy, which neither VHD nor VDI provides, or asking the user, which probably isn't reliable either. On the other hand the VHD and VDI header data structures are very simple, I wouldn't have thought they were very easy to corrupt. I believe also that VBox uses "write through" mode when writing to a virtual disk file on the host, so host caching should not be an issue.

Do you have a more specific form of corruption in mind?

(the only "common" VDI corruptions I'm aware of is when the user has been using snapshots, and not backing up / restoring the right files).

Re: VDI / VHD Repair Tool

Posted: 2. Oct 2009, 14:59
by mark rumsey
I'm guessing here, but I suspect what ProVega is referring to is corruption of the VHD or VDI file itself caused by a problem with the host, which roughly equates to a hardware fault on a physical disk. Whilst some of these faults will clearly trash the file, others may simply prevent it being mounted by Virtualbox, and these are the ones that are most annoying. At the moment any fault with a file that stops it being mounted means the data within is lost, but if there was a way to repair the drive enough to mount it, there is a chance some of that data can be retrieved. Faults to the drive/data structure inside the file are normally going to be caused by and repaired using the guest OS's drive/disk utilites.

One feature that would be useful is the ability to enable the VHD or VDI file to be mounted by virtualbox or the host OS whilst the guest is not running. This feature exists within Windows7/Windows VPC and I found it very useful following a catasrophic VirtualXP crash that caused it to go into a reboot loop that could not be broken. Things were so bad I ended up having to delete and reinstall VirtualXP. However, as I was able to mount the VHD file in Windows 7 I was able to retrieve my data before deleting the VHD and starting again. As the Windows 7 solution works at a low level (the VHD is attached as a virtual device in the disk manager) and that wouldn't be a readily available or portable solution for Virtuabox, perhaps the simplest way to achieve this would be to create a tiny preconfigured guest that is able to connect to the VHD or VDI file and present it as a network drive/share to the host.

Re: VDI / VHD Repair Tool

Posted: 2. Oct 2009, 19:55
by mpack
mark rumsey wrote:Whilst some of these faults will clearly trash the file, others may simply prevent it being mounted by Virtualbox, and these are the ones that are most annoying.
Can you point to a thread where something like that happening is being discussed? I'd like to understand the nature (and cause) of the corruption before I would speculate on how it might be repaired, or whether VBox or any other tool would be able to mount the affected virtual disk.
mark rumsey wrote:One feature that would be useful is the ability to enable the VHD or VDI file to be mounted by virtualbox or the host OS whilst the guest is not running. This feature exists within Windows7/Windows VPC and I found it very useful following a catasrophic VirtualXP crash
By "mounted" I assume you mean with the ability to see and retrieve files. Bear in mind that Win7/VPC only supports Windows OS guests on Windows OS hosts. That means that supported guests use filesystems already supported by the host. Consider the implications of removing those assumptions.

Re: VDI / VHD Repair Tool

Posted: 20. Oct 2009, 00:02
by ProVega
I'm guessing here, but I suspect what ProVega is referring to is corruption of the VHD or VDI file itself caused by a problem with the host, which roughly equates to a hardware fault on a physical disk.
Pretty close - This happened just again today. I have a base VHD and a snapshot... I tried to mount a USB drive and VirtualBox froze (64-Bit host, 64-bit guest Windows 2008 R2, single CPU). I let VBox "think" at 100% CPU for 5 minutes and then the screen stopped painting and windows asked me to close VirtualBox. I did. My host OS was stable, only VirtualBox "crashed" while it did this, it corrupted the "header" of the snapshot and I could no longer mount it with that wonderful "invalid header" error.

Why the HELL is VirtualBox messing with the header anyway and why can't a VHD "survive" a crash of VirtualBox host. It just seems to be a fatal design flaw that if "anything goes wrong on the host, we corrupt the drive and make it unmountable". I am suggesting that there either be a tool to make a drive mountable again, or a design change to VBox to be very careful about writes or commits it makes to the headers (or whatever is causing this error) so that it is more durable in these scenarios. I have had Microsoft Virtual PC crash on me 100's of times too, but it only corrupted the VHD once... VBox has crashed on my 3 times and has corrupted the VHD all 3 times!!!!

Re: VDI / VHD Repair Tool

Posted: 20. Oct 2009, 19:40
by ProVega
VERR_VD_VHD_INVALID_HEADER - And there it is again. VirtualBox hanged for 10 minutes) while I tried to add a shared folder, close program... drive is corrupt.

Lame. :shock:

Re: VDI / VHD Repair Tool

Posted: 20. Oct 2009, 19:41
by ProVega
Oh great, no only did it corrupt the current snapshot, it corrupted the next snapshot down too! The base is still fine though, but that means I loose 1 day of work.

Re: VDI / VHD Repair Tool

Posted: 20. Oct 2009, 21:04
by sandervl
I suggest you open a ticket in our bug tracker. I've had VirtualBox crash on me a million times (quite normal while developing, mind you) without losing the disk image or snapshot. I hardly ever use VHD images though, so perhaps there's a serious problem in that backend.

Re: VDI / VHD Repair Tool

Posted: 21. Oct 2009, 14:14
by mpack
sandervl wrote:I hardly ever use VHD images though, so perhaps there's a serious problem in that backend.
Total guess: perhaps "invalid header" actually refers to an invalid footer, i.e. the idiot decision some developer made to put a "header" at the end of the VHD file, so it has to be moved every time the file grows. Obviously, if the enlarge+newfooter sequence was interrupted that would leave a dead footer.

@ProVega: your first post referred to VHD and VDI corruption. Was that just loose talk, or have you actually seen this kind of corruption happening with VDIs too? VDIs don't have footers, so can't be affected by the issue in the previous para.

Re: VDI / VHD Repair Tool

Posted: 21. Oct 2009, 14:20
by sandervl
mpack wrote:
sandervl wrote:I hardly ever use VHD images though, so perhaps there's a serious problem in that backend.
Total guess: perhaps "invalid header" actually refers to an invalid footer, i.e. the idiot decision some developer made to put a "header" at the end of the VHD file, so it has to be moved every time the file grows. Obviously, if the enlarge+newfooter sequence was interrupted that would leave a dead footer.
That indeed explains a lot. I have to check if we do lazy header/footer updates.

Re: VDI / VHD Repair Tool

Posted: 21. Oct 2009, 21:33
by mark rumsey
Pretty close - This happened just again today. I have a base VHD and a snapshot... I tried to mount a USB drive and VirtualBox froze (64-Bit host, 64-bit guest Windows 2008 R2, single CPU). I let VBox "think" at 100% CPU for 5 minutes and then the screen stopped painting and windows asked me to close VirtualBox. I did. My host OS was stable, only VirtualBox "crashed" while it did this, it corrupted the "header" of the snapshot and I could no longer mount it with that wonderful "invalid header" error.
ProVega, I am only guessing here, but I think the reason for your crash might be down to Windows 2008 R2. Microsoft are making a big thing about the virtualisation side of this new release, and I suspect that R2 might be looking to enable hardware virtualisation during start up, and that would almost certainly kill VirtualBox. Like I said, I'm guessing and I could be totally wrong, but being a bit of a cynic I would not be surprised to find Microsoft have deliberately created an operating system that cannot be run in a virtual environment.

Re: VDI / VHD Repair Tool

Posted: 24. Oct 2009, 21:18
by ProVega
Thanks for the thoughts

1.) This is with VHDs only... if there is something inherently safer about VDI's I would love to know. If I just cover a VHD to a VDI do I get the benefit?

2.) My guest is Windows 2003 R2 and 2008 R2... so both have problems. This seems to be a host problem though, not a guest problem.

3.) This still seems to be a design flaw... or something - there seems to be no excuse to have this level of data loss when the host process crashes (for whatever reason)

Re: VDI / VHD Repair Tool

Posted: 24. Oct 2009, 21:39
by Perryg
@ProVega,
Did you open a Ticket as suggested?

Now for the question of whether a VDI is better than VHD?
I believe there is. IMHO it runs faster and is less likely to get corrupted. But that is just my opinion. I run VDI, VHD,& VMDK and the VDI files seem to last longer and run faster for me.

Re: VDI / VHD Repair Tool

Posted: 28. Oct 2009, 10:22
by sandervl
Should be fixed in 3.0.10. We were indeed updating the footer in a lazy way. If the VM process crashed (which shouldn't happen in the first place), the VHD could be left in an inconsistent state.

Re: VDI / VHD Repair Tool

Posted: 28. Oct 2009, 14:01
by mpack
Thanks for letting us know.