CloneVDI tool - Discussion & Support

Discussions related to using VirtualBox on Windows hosts.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

I think the readme is quite clear :-

CloneVDI does not support resizing of any Linux filesystem (unless of course you consider FATx or NTFS to be Linux filesystems). It can still move partitions aside to make room for a larger main partition (i.e. leaving the new unallocated space immediately after the main partition), and then yes, you would use GParted to do the final bit. This limitation is not because CloneVDI doesn't recognize the EXTx filesystem: of course it does. It's just that EXTx resizing is messy and complicated and not something I had time to work on. The other filesystems make it much easier.

CloneVDI does support compaction of Linux filesystems i.e. (EXT2/EXT3/EXT4). Therefore you don't need to use zerofree to indicate the unwanted areas.

All of the above is with the proviso that CloneVDI can find the partitions, and since it only understands MBR partition maps this excludes people who use LVM and GPT.
StoneThrower
Posts: 2
Joined: 31. Aug 2017, 16:35

Re: CloneVDI tool - Discussion & Support

Post by StoneThrower »

Hi there, mpack

Having finally conceded I could no longer dedicate time to scratching around deleting logs and other accumulating data within an absolutely-full, but much-valued, fixed-size VDI, I used your tool today to increase the size of the disk. Your tool worked like an absolute dream. It took just a few minutes and 5 GB is now added and space will never be an issue again. Hours will be saved in the years ahead. Thanks so very much. I imagine I am just one of thousands of users who have benefited from your work here.

I am not a VM administrator, I am really a novice, but I would like to ask one question to you or the wider forum readership if I may. When I created the original VDI many years ago, I expect my motivation for creating a fixed-size disk was to increase performance. My understanding is that the new disk created by your tool is dynamically-sized, which *could* have negative performance implications, but maybe not these days, because the virtual disk is residing on my PC's SSD - and my understanding of SSD's is that fragmentation is less of an issue. So to the question - is there any *meaningful* performance advantage in reverting the new disk back to a fixed size?

And one final time, thanks for the tool. 8 years after your original posting, and you're still receiving applause!
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Thanks for the feedback.
StoneThrower wrote:I expect my motivation for creating a fixed-size disk was to increase performance. My understanding is that the new disk created by your tool is dynamically-sized, which *could* have negative performance implications
That was always an urban legend, nothing more. Lots of (uninformed) people keep repeating the legend, but not one person has ever produced a benchmark to prove that it's the case, nor offered a logical argument (based on reality, not on more legend) as to why it should be the case.

The actual fact: dynamic VDI and fixed size VDI are exactly the same file format, so there can't possibly be any feature inherent in the format which would explain a performance difference. The only real difference is that one of these takes more up front disk space on the host, which ironically tends to make people under specify the disk size, leading to guest congestion and poor performance. And with the world going SSD even the ill informed assumptions about seek times which justified the initial decision stop making any sense at all.

My advice to you: use dynamic VDI for now. Start worrying about the loss of performance after you notice any. My expectation is that you won't, for reasons given.
StoneThrower
Posts: 2
Joined: 31. Aug 2017, 16:35

Re: CloneVDI tool - Discussion & Support

Post by StoneThrower »

mpack wrote:My advice to you: use dynamic VDI for now. Start worrying about the loss of performance after you notice any. My expectation is that you won't, for reasons given.
And that is advice I am grateful to receive and will happily take!
IzK6666
Posts: 2
Joined: 8. Aug 2016, 21:34

Re: CloneVDI tool - Discussion & Support

Post by IzK6666 »

Great utility. I was using Virtualbox for a few years and this tool is just what I missed from VMWare: a real compact function!

As my goal was only to compact vdi files, I added the option into the right click button but not as default option to avoid accidental clicks.
If someone is interested, copy the attached file in the same folder as CloneVDI.exe, change the extension to BAT and run it.
Tested in WXP and W7.

Edit: File modified. Now works under W10 too. Remember to run the script as administrator.

Mod note: WARNING: I feel it important to warn people that running the attached text file (as a bat file) causes it to make changes to the Windows registry on your host, to add the right click context menu functionality described.
Attachments
RightClick.txt
(515 Bytes) Downloaded 108 times
RightClick.jpg
RightClick.jpg (4.84 KiB) Viewed 9815 times
Last edited by mpack on 26. Sep 2017, 19:38, edited 2 times in total.
Reason: Add mod note.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Thank you, and I'm impressed! Some arcane knowledge of shell, registry, and batch file tricks on display there.
Kumba
Posts: 43
Joined: 2. Aug 2011, 05:48
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Win311, Win98, NT4, Win2k, Win2k3, Win2k12R2, OpenBSD, MS-DOS 7.10, NW312, NW42SP9, NW65SP8, Haiku

Re: CloneVDI tool - Discussion & Support

Post by Kumba »

This question might have been answered already, but it's difficult to rummage through 76 pages of a thread when you're not certain about the right keywords, and Google's "fuzzy" search habits don't help things much, either.

I am looking for information on converting VMDK disk images that include a VMDK snapshot image, all split up into 2GB sparse files, into a single VDI or at least a VDI with a single snapshot. So far, I can easily convert the base VMDK to VDI using either VBoxManage or CloneVDI. CloneVDI also successfully converts the snapshot into a ~2MB VDI, which suggests all I have is an empty snapshot and can thus ignore it. But I want to be sure, because I have three of these things to convert and haven't yet looked at the other two.

Assuming CloneVDI converts both base and snapshot VMDKs to VDIs, I am not seeing a very straightforward way to "link" the snapshot VDI to the base VDI. It is suggested via these forums that the snapshot should have the parent's UUID encoded in the VDI header someplace. I suppose I could edit that in with a hex editor, but that seems too easy. There's got to be more involved (because nothing is ever easy or straightforward with computers).

Or is this all a non-issue and CloneVDI somehow can detect the VMDK snapshot and merge it into the base VMDK and then write the whole thing out to a single VDI?

Edit: Figured things out enough. The VMDK snapshots were just empty snapshots in the end. But to be extra sure, I still re-linked the converted VDI's into the parents using "vboxmanage internalcommands" and the "sethduuid" and "sethdparentuuid" subcommands to assign UUIDs and whatnot, then test-booted to be really sure. All three VMs come up just fine for my needs. I then merged the converted snapshots into the base image and created new ones, so this should solve the problem. Someone really needs to make a GUI for all of the little tweaks that vboxmanage has available. Talk about an overloaded executable...
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

I'm not sure I understand what you were doing with sethdparentuuid etc. The VDIs created by CloneVDI are always stand alone. They have no parents or children, and forcing them to contain non-zero (lying) header fields can only cause problems.

If you cloned a snapshot VMDK using CloneVDI, and if CloneVDI said it succeeded, then it should have created a merged VDI image having made use of the base VMDK and all snapshots in the chain. That said, there are so many variants of VMDK knocking around (even UUIDs are optional) that it's possible that without the control file you couldn't tell it was a snapshot. VirtualBox doesn't create VMDKs like that, and VirtualBox (not VMWare) is the focus of CloneVDI.
dainn
Posts: 2
Joined: 25. Oct 2017, 16:42
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: Mostly Windows 7

Re: CloneVDI tool - Discussion & Support

Post by dainn »

Thanks for putting this tool out there mpack. I do have a couple questions for you, and please forgive me if they seem common sense; I am very new to the VM world.

In the initial post, you state CloneVDI has:
Ability to repair common forms of header corruption.
I'm running a Ubuntu box on VirtualBox 5.1.26 and after having to make a hard shutdown due to my system freezing the VM won't spin up, resulting in the following error:
Stderr: VBoxManage.exe: error: Could not open the medium '<path/filename>.vmdk'.
VBoxManage.exe: error: VMDK: inconsistency between grain table and backup grain table in '<path/filename>.vmdk' (VERR_VD_VMDK_INVALID_HEADER).
VBoxManage.exe: error: VD: error VERR_VD_VMDK_INVALID_HEADER opening image file '<path/filename>.vmdk' (VERR_VD_VMDK_INVALID_HEADER)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component MediumWrap, interface IMedium
I'm hoping this tool will help in recovery (it has been indicated that it has done so for similar issues in some of the threads I found while looking for a fix). My questions for you are:
1) Since I will not be looking at expanding or compacting the drive, the fact it is in Linux LVM format should not pose an issue, correct?
2) Since this will be moving from .vmdk to .vdi, I would want to run with a new UUID, correct?
3) Once I make the clone, assuming it is successful, will I need to create a new machine or can I point the current machine to the new .vdi file?

Thanks!
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Sorry, but the header repair feature only applies to the VirtualBox hdd format, i.e. VDI, not VMDK.

But, you can still test whether CloneVDI will clone the VMDK for you - which of course means that if it succeeds then you'd have a VDI with a good header.

Yes, LVM is only a problem if CloneVDI needs to analyze the filesystem, which is not required for simple cloning.

You can keep the UUID when cloning from VMDK->VDI, that isn't a problem, and prevents certain consequences e.g. with grub.

If the original machine doesn't use snapshots etc and you keep the UUID then the VDI ought to be a plug in replacement for the VMDK - which of course must be unregistered first.
dainn
Posts: 2
Joined: 25. Oct 2017, 16:42
Primary OS: MS Windows 7
VBox Version: OSE other
Guest OSses: Mostly Windows 7

Re: CloneVDI tool - Discussion & Support

Post by dainn »

Thanks for the quick response! I will give it a go. Hopefully I will end up with a .vdi file that works. :) I'll let you know one way or the other (and also as a reference point for others who may find themselves in a similar situation).
The original machine was not utilizing snapshots, so I'll cross my fingers for an easy swap if the clone works.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

dainn wrote:I'll cross my fingers for an easy swap if the clone works.
I would suggest backing up the VM files, just in case you want to go backwards. Of course you can omit the VMDK from the backup.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

NEW VERSION RELEASED - v3.01 (see root message for download link).

This is probably the most minor release I've ever done. The main purpose of this release is to tweak the header repair feature to help a particular user.

The only other change is that I've been refreshing all the icons in my older apps, including CloneVDI, using new vector icons created in InkScape (a marvelous tool). The new icons go up to 256x256 size and so look better on hidpi displays, which is where I assume we're all heading very soon.
sdrubble
Posts: 15
Joined: 19. Jul 2009, 23:07
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Win XP, Linux Mint
Location: Belo Horizonte, Brazil

Re: CloneVDI tool - Discussion & Support

Post by sdrubble »

mpack wrote:NEW VERSION RELEASED - v3.01
Thank you very much for the new release.

For some odd reason (it might have to do with your compilation environment) the timestamps on the .exe and the .txt files within your 3.01 zip package are exactly the same as they were within the previous 3.0 package (July 5, 2017). Contents and filesizes of these files are obviously different between releases.

I just have the old habit (OCD ?) of checking timestamps on newer versions of stuff, coupled with these other even older habits of keeping previous versions of installer packages and actually doing comparisons. It's sometimes mildly useful... :mrgreen:

That's about it - just wanted to let you know.

Cheers, and happy New Year ! :D
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Hmm, thanks. It must be a side effect of how I created the zip. I'm always afraid of leaving out an important file, so what I do is copy the previous zip, rename it, and then I replace each of the files, if they've changed (e.g. the CloneCLI.bat file has so far never needed to change).

In the past I always used WinZip to manage zip files, but this time I used the built in Win10 "Compressed Folder Tools" feature, and dragged the changed files onto the zip window. I guess that the Windows tool doesn't change the timestamp when replacing files in a zip.

What tool did you use to check the dates?
 Edit:  That's really weird. If I view the zip in WinZip then it shows what you report. If I view it using Windows native zip viewer then each file shows the correct dates. This is with a fresh zip I just downloaded from the root message, so Windows can't be using some kind of external metadata. But, presumably the zip unpacked ok for you, so it can't be using an incompatible header either. Dirty tricks afoot...

[Edit2]
Still weird. It happens that I have a spec for the original ZIP format, so I just looked at the zip headers in a hex editor. Basically a zip is a bunch of files concatenated together into a stream, each with a local file header (a bit like a tarball). Then there's a central directory tacked on the end. Both the local header for CloneVDI.exe and the central directory both have the same date for the file, and it's correct: 24th Dec. I'm going to park this now. I'm going to guess that there has been sneaky changes to the zip format by WinZip, but Windows still goes by the original spec. 
Post Reply