update OS -> corrupt VDI?

[MountainTrails]

I recently updated my system from OpenSUSE 13.1 to 13.2

The VirtualBox files were in an untouched partition.

I had 2 virtual machines. One is working; the other (the critical one) is giving this error:



Failed to open a session for the virtual machine Windows 8.1.

Could not open the medium '/home/USERNAME/VirtualBox VMs/Windows 8.1/Windows 8.1.vdi'.

VDI: invalid header in '/home/USERNAME/VirtualBox VMs/Windows 8.1/Windows 8.1.vdi' (VERR_VD_VDI_INVALID_HEADER).

VD: error VERR_VD_VDI_INVALID_HEADER opening image file '/home/USERNAME/VirtualBox VMs/Windows 8.1/Windows 8.1.vdi' (VERR_VD_VDI_INVALID_HEADER).

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: MediumWrap
Interface: IMedium {4afe423b-43e0-e9d0-82e8-ceb307940dda}



I have done some googling/reading and none of the various approaches seem to have worked.
I would very much like to recover a couple files from the .vdi,

Any help or suggestions would be very appreciated.

[mpack]

I have done some googling/reading and none of the various approaches seem to have worked.

Which approaches were those? I'm not aware of any generic approaches that could be used for this, except of course to restore the VDI from a backup. Using the wrong procedure might mess up the data further.


Use a hex editor such as frHed to partially open the VDI, save the first 2MB (2097152 bytes) to a bin file, zip the bin file and attach the zip here.

I will also require the exact logical size of the drive, and the current exact VDI file size, to the nearest byte.

[MountainTrails]

I have done some googling/reading and none of the various approaches seem to have worked.

Which approaches were those? I'm not aware if any generic approaches that could be used for this, except of course to restore the VDI from a backup. Using the wrong procedure might mess up the data further.

I stayed vague because clearly I'm no expert here, and when I troubleshoot someone else's issues I like to come in clean with my approach. I monkeyed with various tools that would mount or try to read the file, and they were passive. I can go down the list, but . . .

Use a hex editor such as frHed to partially open the VDI, save the first 2MB (2097152 bytes) to a bin file, zip the bin file and attach the zip here.

I will also require the exact logical size of the drive, and the current exact VDI file size, to the nearest byte.

The requested file is attached.
The exact VDI filesize is 917610496 bytes.

The exact logical size of the drive -- hmm. In the virtual machine's Settings->Storage view, it shows:

Virtual Size: --
Actual Size: --
Details: Dynamically allocated storage

[mpack]

This is awkward. This VDI must be quite old - it's using 4K alignments in the header. That means that you can't use VirtualBox to create a replacement header. Do you have any backups of this VDI at all?

As to the damage, the VDI header seems to have been largely zeroed, but not the preheader. Most odd. Did you mess with the preheader perchance?

[MountainTrails]

This is awkward. This VDI must be quite old - it's using 4K alignments in the header. That means that you can't use VirtualBox to create a replacement header. Do you have any backups of this VDI at all?

That is very strange. The VDI was created in the summer of 2015. I was running openSuSE 13.1 at the time.

No, no backup. And yes, I am a moron. Backing up my laptop was on the "to do" list, and this was the only important thing on it.

As to the damage, the VDI header seems to have been largely zeroed, but not the preheader. Most odd. Did you mess with the preheader perchance?

No, not knowingly.

As I said, I reinstalled an OS. The existing /home partition was not touched or modified. When the new system was installed, there was a new home dir, $NEWHOME. I did mount the old home partition and do this:

cp $OLDHOME/blahblah/guestOS.vdi $NEWHOME/blahblah/guestOS.vdi

[mpack]

Well, I'm done for the day now, but I will tell you that the block map looks intact, meaning that the disk is almost certainly recoverable. I'll find a way to make a replacement header tomorrow morning.

[mpack]

Ok, here is your fix.

1. Back up your existing VDI so you can't make things worse.
2. Download the attachment, unpack the zip, and use FrHed to replace the first 512 bytes of your broken vdi with the replacement header I've provided.
3. Save and close FrHed.
4. This step is optional: correct the UUID in the VDI header, discussion below.

I didn't know your old UUID, so the UUID in the new header is different. If you want to use the old UUID then you first need to find out what it should be - you can see that in the media registry of the .vbox file. You can then use "VBoxManage internalcommands sethduuid "Windows 8.1.vdi" <old uuid>" to set the replacement UUID.

Alternatively, just release and remove the old media, and use the VM storage panel to add the new media. However, Windows 8.1 might want to reactivate because of the media signature change.

[MountainTrails]

First of all, thanks. Your response already has been above and beyond anything I had hoped for.

I replaced the first 512 bytes of the VDI with the material from your file. FrHed is WIndows; I'm sitting at a Linux host, so:

dd if=New\ Header.bin of=Windows\ 8.1.vdi count=512 bs=1 conv=notrunc

I fixed the UUID mismatch with your "VBoxManage internalcommands sethduuid ..." command. VirtualBox gets started and seems happy. I go to start the virtual machine and . . .

now we're here, at a black screen. From the VBox.log:

Out of range access (7718563840) in image

[mpack]

Hmm. That implies that there's a problem with the block map, indexing a block which doesn't exist. The block map looked credible to me, but I didn't check every index. I'll have a look at that tomorrow. If blocks have been lost then there's going to be corruption of the image.

In the meantime I would suggest backing up the patched VDI (again) and then try mounting it as the second drive in another VM. You might be able to get important files out.


Another thing I'd be doing if I was you is check the disk and filesystem integrity on whatever medium was responsible for storing this VDI, because this is looking less and less like any kind of VBox glitch. I've never had any such glitch in 7 years - and you've got two different ones on the same file? No way.

[mpack]

Hmm. That implies that there's a problem with the block map, indexing a block which doesn't exist.

Unfortunately, it turns out to be much worse than that. Most of the block map seems to refer to blocks which don't exist.

To explain the numbers: you told me above that the current VDI file was 917610496 bytes (please confirm). Subtract a combined header and block map of 106496 bytes (easily visible in the 2MB provided), that leaves exactly 875MB of image data, i.e. 875 allocated 1MB blocks. Less than 1GB. I believed it likely this was a complete image because it is an exact number of MB despite a non integral header size. So, I synthesized a new header for you which indicated a drive with 25600 blocks total (25GB), 875 allocated.

However, detailed examination of the block map indicates that in fact it indexes 25451 individual allocated blocks, so this file must have been truncated, losing 24576MB of data, i.e. 96% of the data from a nearly full 25GB drive has been discarded. There's no way to recover from that.

I'm having a great deal of trouble understanding how this happened. Did you do anything you didn't tell me about, such as trying to enlarge the drive with the wrong tool? I don't suppose it's coincidental that the virtual drive was nearly full.

[MountainTrails]

The whole thing is incomprehensible to me. When I installed the new OS version, it did not touch the partition where these vbox files are.

> ls -l *vdi
-rw------- 1 me users 917610496 Oct 17 16:42 Windows 8.1.vdi

As far as ANYTHING that could have affected the file -- and this seems far-fetched -- I run an encrypted home directory:

# df -BM
Filesystem             1M-blocks    Used Available Use% Mounted on
/dev/sda6                 20031M   6432M    12559M  34% /
devtmpfs                   5858M      0M     5858M   0% /dev
tmpfs                      5866M     17M     5849M   1% /dev/shm
tmpfs                      5866M      3M     5864M   1% /run
tmpfs                      5866M      0M     5866M   0% /sys/fs/cgroup
/dev/sda7                107920M 101748M     5169M  96% /home
/dev/mapper/_dev_loop0    98300M  50115M    43186M  54% /home/me

# badblocks -v /dev/sda7 
Checking blocks 0 to 112405503
Checking for bad blocks (read-only test): done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)

# badblocks -v /dev/mapper/_dev_loop0
Checking blocks 0 to 102397951
Checking for bad blocks (read-only test): done                                                 
Pass completed, 0 bad blocks found. (0/0/0 errors)

If you're interested in digging in for any further analysis, even just your own curiosity, I can make the entire VDI file available (privately). There are no nuclear launch codes on it.

But, it seems I have a long tedious road ahead in trying to reconstruct the one thing I need. One file. Damn, the laptop is backed up now, but that oversight was a lesson I learned a long time ago, and apparently forgot.

[mpack]

I'm sorry, but I feel that I've completed my analysis: the file has been truncated to around 5% of its previous size. That is fatal. It makes it pretty certain that the internal guest OS had been destroyed, and possibly the filesystem too.

I had originally intended to patch the block map to discard the out of range blocks. I expected corruption of course, but when I discovered the extent of it, a patch seemed pointless. However, I've attached it anyway in case you believe in miracles. Use the attached binary to overwrite the first 106496 bytes of your VDI. This patch includes both the new header and the new blockmap. The UUID will be wrong, but who cares now.

You can see if this boots, but I doubt it. You can see if it's accessible when attached to another VM. Perhaps you have a better chance with that. You can examine the drive contents with a sector viewer. Good luck.

[MountainTrails]

No success with the overall situation, but I am grateful for the attention you gave this thread. Thank you, sincerely. At least I know it's not just my inexperience with virtualbox internals that is the limiting factor.

I'm going to give recovery of the one critical file a serious effort. I DO believe! I DO! I ran the strings command on the VDI file with grep for a text string unique to the content of that file (it is a Framemaker document), and IT IS IN THERE.

I may suck at disk forensics, but I'm taking a run at this.

[mpack]

As I said, check and see if you can mount the patched VDI in another VM. The first thing that happens to a new VDI is that the filesystem is created, which tends to ensure that the filesystem is contained within the first few blocks of the VDI, so while the guest OS is likely to be scattered everywhere (so you can't boot from it), there's a chance (not a good one) that the filesystem is intact, so it may mount as drive2 in another VM, and you might be able to copy your file off.

Only if that fails would I resort to scanning for file fragments, at least not unless the file is less than 4K (i.e. fits into one NTFS cluster).

[mpack]

There is one other thing you can do once the patch has been applied, which is to clone the drive using CloneVDI. The effect will be to rearrange the image blocks into linear order, which may make it easier to find file data by scanning. Cloning with VBoxManage might do the same, I'm not sure.