VMDK: inconsistency between grain table and backup grain table

Discussions related to using VirtualBox on Windows hosts.
Locked
Gamesntoys
Posts: 4
Joined: 2. Mar 2017, 11:55

VMDK: inconsistency between grain table and backup grain table

Post by Gamesntoys »

Hi,

I'm a VirtualBox novice who has used the program for very basic uses a few times over the years without incident.

This week I had need to make a LOT of virtual machines and everything worked fine for a few days until I ran into this error (on one of over a hundred VMs) shown in attached screenshots. I did not have a back up of most of the VMs (or the one with the error) since this was all very experimental, anyway. The loss of this one VM wouldn't be the end of the world, but I'm concerned that the same thing might happen again and just want to know how to deal with it. (I believe the corruption resulted from a power outage yesterday morning, but I'm not certain that specific VM (#138) was running at the time or that the power outage was the cause.)

From the error, "VD: error VERR_VD_VMDK_INVALID_HEADER opening image file" it seems that the error is in the header of the VMDK file, which may be "treatable." I've learned from these forums that the VMDK filetype is not the VD of choice for VirtualBox and that support for VMDK files is a little less extensive than for VDI files.

I've also learned from this thread (that I apparently can not link to since I am a new member, Forum #6, Topic #70007, entitled "[Solved] Failed to start virtual machine. please help me!") that fixing the headers of a virtual disk file may not be that complicated, though that thread pertains to working with VDI files. Since I'm not an expert on this whatsoever, I was hoping for some insight on how to proceed. Thank you very much for any assistance or insight you can share.
Attachments
Screenshot - 3_2_2017 , 04_52_00_ver002.png
Screenshot - 3_2_2017 , 04_52_00_ver002.png (40.78 KiB) Viewed 24985 times
Last edited by mpack on 2. Mar 2017, 12:52, edited 1 time in total.
Reason: Add link
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VMDK: inconsistency between grain table and backup grain table

Post by mpack »

Grain table (same as "block map" in VDI files) errors are not easily fixable, at least around here.

This type of error could only have been caused if the VM was running when the corrupting event (e.g. the power outage) occurred. It means exactly what the error says: the VMDK has a primary and backup grain table, the first of which is updated frequently as the disk is written to, the second is updated on a lazy basis - and force flushed when the VMDK is closed. Hence, a difference means that the VMDK was never closed.

Yes, use VDI if you want repair help here (and in fact that particular error condition can't arise in VDI). For VMDK you might find repair help on a VMWare forum, though as a non-VMWare user I don't know how much welcome you'll get.

Re the thread you wanted to link to (I added the link): yes, repairing a VDI header can be easy. Unfortunately you are using VMDK, not VDI, and the damage is to your grain table, not to some unimportant signature block at the start of the file! So, the other thread is not really a similar case.
Gamesntoys
Posts: 4
Joined: 2. Mar 2017, 11:55

Re: VMDK: inconsistency between grain table and backup grain table

Post by Gamesntoys »

Your assistance (and fixing the link) is much appreciated. Evidently, I was overly hopeful that it was a simple problem due to the "VERR_VD_VMDK_INVALID_HEADER opening image file" error. Too bad it's more complicated than the simple fix in the other thread. I'll let this serve as a lesson to be more cautious with VMs in future inclement weather. Thank you.

Additionally, do you recommend any resource for learning the ins and outs and differences of the VDI and VMDK filetypes? I simply used VMDK because I've heard of it more often.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: VMDK: inconsistency between grain table and backup grain table

Post by mpack »

Gamesntoys wrote: Additionally, do you recommend any resource for learning the ins and outs and differences of the VDI and VMDK filetypes? I simply used VMDK because I've heard of it more often.
Other than studying the VirtualBox source code, and the VMDK spec (which is what I did), no I don't know of anything.

There is an old-but-good overview of the VDI format here, written by a former moderator: viewtopic.php?f=35&t=8046. Ignore discussion of non optimal sector alignments, as that was fixed years ago. VDI currently uses a 1MB alignment, which is good for a modern OS.

The golden rule to stay out of trouble is: stick to dynamic VDI. Don't use other formats, don't use fixed size disks. Back up the VM folder if you don't want to lose it, and avoid snapshots (snapshots are fragile, snapshots are NOT backups). Also, make the dynamic VDI a healthy size - neither too big nor too small. I typically find 32GB to 64GB to be ideal, assuming of course that I'm not intending to load up a whole lot of stuff in any one VM.
Gamesntoys
Posts: 4
Joined: 2. Mar 2017, 11:55

Re: VMDK: inconsistency between grain table and backup grain table

Post by Gamesntoys »

I fixed the VMDK and will explain how if it's of any help to anyone else at some point:

So I Googled a little and found out that VMWare has a similar command line disk manager to VB's. It happens to have a repair functionality for VMDK files (overview of which can be found by googling "Repairing a virtual disk in Fusion 3.1 and Workstation 7.1 (1023856)"). I installed VMWare, and made a VM to figure out where it puts the files, then copied the corrupted VMDK from the Virtualbox VMs folder in Users to the location of VMs in VMWare (in the users documents folder or in my case D:\Documents\Virtual Machines\Windows 7\). I tried to "repair" the trial VMDK I just created with VMWare and was told it had no errors. Then I ran the repair option on the corrupted VMDK, and it was repaired, as shown in the pic.

I copied the VMDK back to where it belonged and restarted the gui (Oracle VM Virtualbox), the VM showed as "Normal" (instead of "Differencing, Inaccessible). So I ran the VM, and it worked, going to the default 30 second timer asking if I wanted to start Windows in Safe Mode. I elected to Start Windows Normally, and it started normally.

Using vmware-vdiskmanager from command line to repair the corrupt VMDK file:
Screenshot - 3_2_2017 , 06_48_08_ver001.png
Screenshot - 3_2_2017 , 06_48_08_ver001.png (23.59 KiB) Viewed 24967 times
Before:
Screenshot - 3_2_2017 , 04_50_59_ver003.png
Screenshot - 3_2_2017 , 04_50_59_ver003.png (7.66 KiB) Viewed 24967 times
After:
Screenshot - 3_2_2017 , 07_15_25_ver001.png
Screenshot - 3_2_2017 , 07_15_25_ver001.png (6.88 KiB) Viewed 24967 times
Last edited by Gamesntoys on 2. Mar 2017, 15:00, edited 2 times in total.
Gamesntoys
Posts: 4
Joined: 2. Mar 2017, 11:55

Re: VMDK: inconsistency between grain table and backup grain table

Post by Gamesntoys »

mpack wrote:There is an old-but-good overview of the VDI format here, written by a former moderator: link was here. Ignore discussion of non optimal sector alignments, as that was fixed years ago. VDI currently uses a 1MB alignment, which is good for a modern OS.

The golden rule to stay out of trouble is: stick to dynamic VDI. Don't use other formats, don't use fixed size disks. Back up the VM folder if you don't want to lose it, and avoid snapshots (snapshots are fragile, snapshots are NOT backups). Also, make the dynamic VDI a healthy size - neither too big nor too small. I typically find 32GB to 64GB to be ideal, assuming of course that I'm not intending to load up a whole lot of stuff in any one VM.
Thanks for the link, hopefully it will tell most of what I need to know. I cloned over a hundred VMs with VMDK, while the main VM is dynamic, set for max size of 120GB. A bit large I guess but oh well, it actually takes up less than 20GB at this point, and each of the clones take up from 200MB to 800MB, so space doesn't seem to be a major issue). I'll keep them for now but in the future I'll do as you suggest. I have these VMs located on an SSD, and will be backing them up to a normal large hard drive (once a week is often enough). Is that an inherently bad idea? I just want the VMs to launch and run as quickly as possible, but not sure how much it will stress the drive to have ~160 VMs loading up and running (individually) for a few minutes each on a schedule each day?

I really appreciate your assistance! Thank you again!
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: VMDK: inconsistency between grain table and backup grain table

Post by scottgus1 »

First, regarding backups, see: Moving a VM and re-interpret it as "Backing Up a VM".

Since you are using what appears to be linked clones, be sure you back up everything, including the source disk.

Each clone may get as big as or bigger than the original source disk, by the way. Be prepared for running out of host disk space. If I recall correctly, the guest goes on pause when the host disk runs out of space, so you can try to do something about the space issue. I have read and use a trick to be able to do something substantial about lack of host disk space quickly: I have a gigabyte of some large files I don't use on the host disk, like a few copies of a 300MB video driver installer file, etc. in a scrap folder on the drive root, so I can find it quickly. If I run out of disk space, I can delete a scrap file or two and instantly gain 300-600MB, giving me space to handle the guest without killing it.

Also, my experience is that three or more modern OS guests booting or heavy-disk-using on a platter drive may and probably will swamp it - the heads can't move around fast enough to satisfy both OS's) and issues may develop, like an OS or two reporting loss of the boot disk & BSODs, etc. Each clone will have to read the base disk, at least, at different times, and read/write their differencing disk, so you are going to have problems running more than two at once. If you run only one of the 160-ish at each time you'll be OK.

One final issue that may develop is that the differencing disks may get so big that you may feel compelled to move them off the host drive containing the source disk. I don't know if this can be done or how. Might be built-in functionality in Virtualbox, might be a bit of simple XML editing in the .vbox file for the clone, might be impossible, I'm not sure.
pooley1994
Posts: 1
Joined: 20. Apr 2018, 04:23

Re: VMDK: inconsistency between grain table and backup grain table

Post by pooley1994 »

I made an oracle account purely to post on this thread. I had the same issue, and the solution worked for me as well.

I had already had VMWare Workstation installed, and so the script "vmware-vdiskmanager.exe" was already present. For me the command looked like the following:

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -R "C:\Users\Fake\VirtualBox VMs\MSEdge - Win10TH2\Snapshots\{492c66e9-eb2a-4129-a1fe-4ca1816817b9}.vmdk"

And the output was essentially the same:

"The virtual disk, 'C:\Users\Fake\VirtualBox VMs\MSEdge - Win10TH2\Snapshots\{492c66e9-eb2a-4129-a1fe-4ca1816817b9}.vmdk', was corrupted and has been successfully repaired."

At this point I was successfully able to start the VM up again in virtual box. Thanks everyone!
Satcom805
Posts: 2
Joined: 16. Aug 2018, 20:48

Re: VMDK: inconsistency between grain table and backup grain table

Post by Satcom805 »

i really need help on fixing this issue on a MAC running High Sierra 10.13.6.


thanks
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: VMDK: inconsistency between grain table and backup grain table

Post by socratis »

@Satcom805
You have two options:
  1. Find a computer that runs Windows or Linux, install VMWare Workstation and try to fix your VMDK.
  2. Install a Win10 (you could download a pre-made one from Microsoft) or a Linux VM, install VMWare Workstation and try to fix your VMDK.
Don't forget to convert your VMDK to VDI once you're (hopefully) done. And, please try to stick to the VirtualBox defaults, if you want to be supported by VirtualBox.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Satcom805
Posts: 2
Joined: 16. Aug 2018, 20:48

Re: VMDK: inconsistency between grain table and backup grain table

Post by Satcom805 »

ok... i will give it a go. wish me luck
merihcobanmc
Posts: 2
Joined: 7. Sep 2019, 17:26

Re: VMDK: inconsistency between grain table and backup grain table

Post by merihcobanmc »

Gamesntoys wrote:I fixed the VMDK and will explain how if it's of any help to anyone else at some point...
That worked for me thanks :)
Locked