CloneVDI tool - Discussion & Support

Discussions related to using VirtualBox on Windows hosts.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

@EthanH:

I've just uploaded a new version of CloneVDI (v4.01) to the first page. Perhaps you could give it a try and report back.

It was actually a combination of two separate crashes, but since both were memory access exceptions you would get the same result code.

Crash #1 (invalid option used): was due to an error when trying to format the error string which says "unknown option". The fact that no error window or message appears ought to have been a clue for me here. Since people generally don't give it invalid command line options this has gone unreported until now.

Crash #2 (-c option used): has to do with the new GPT support features. I had to change the old "open a new disk" code because I need to know quite early on whether I'm dealing with an MBR or GPT drive. I neglected to make the same changes to the command line startup code.
EthanH
Posts: 7
Joined: 15. Nov 2019, 20:12

Re: CloneVDI tool - Discussion & Support

Post by EthanH »

@mpack:

Your fixes worked perfectly. Many thanks!
Crash #1 (invalid option used): was due to an error when trying to format the error string which says "unknown option". The fact that no error window or message appears ought to have been a clue for me here. Since people generally don't give it invalid command line options this has gone unreported until now.
I'm good at creatively breaking things.
EthanH
Posts: 7
Joined: 15. Nov 2019, 20:12

Re: CloneVDI tool - Discussion & Support

Post by EthanH »

I attached a script we use to automate virtual disk compression with mpack's excellent CloneVDI tool. We use VirtualBox extensively to run test and measurement machines. Our typical servers host 8 - 12 VMs. Operating system updates (many of the VMs are Windows), general cruft, etc. bloat the size of the virtual disks over time. Manually compacting these drives is tedious, and while vboxmanage modifymedium --compact works, it requires first defragmenting the VM disk in the guest o/s, and zeroing out free space to work well. CloneVDI is far simpler.

The attached script iterates through all VMs registered with VirtualBox on your computer then calls CloneVDI to compact each vdi disk (the script is hard-coded for vdi; I have not tested CloneVDI on other virtual disk formats). Before compaction begins, any running virtual machines are shut down and VirtualBox is closed.

For full automation, either put the script in the same folder as CloneVDI.exe or pass the path to CloneVDI on the command line. Example:

Code: Select all

(Powershell)
PS .\CompactAllVDI.ps1 "C:\Some Folder\CloneVDI.exe"

(command)
CompactAllVDI.ps1 "C:\Some Folder\CloneVDI.exe"
If CloneVDI is not in the same folder as CompactAllVDI.ps1, a dialog opens prompting you to locate it. The default action is to clone and compact each virtual disk, delete the original to the recycle bin, then rename the clone to the original disk name. VirtualBox is none the wiser, and you are left with vdi files that consume less disk space on your system. Compacting Windows 10 guests once a month or so typically frees 7 - 12GB per virtual disk.

If your VMs live on a SSD, particularly one without significant amounts of free space, the cloning and compaction process can consume enough space that throughput drops significantly. CompactAllVDI.ps1 has a command line option to empty the recycle bin for the drive a VM is on after the clone-compact process completes on each drive. This frees SSD blocks and fires the TRIM command to start garbage collection on the drive. The result is disk throughput stays more consistent. To enable this mode, set the -EmptyRecycleBin to $true:

Code: Select all

PS .\CompactAllVDI.ps1 -EmptyRecycleBin $true
For a fully automated flow (e.g. a scheduled task), launch the script with the path to CloneVDI and, if desired, EmptyRecycleBin:

Code: Select all

PS .\CompactAllVDI.ps1 -PathToCloneVDI "C:\Some Folder\CloneVDI.exe" -EmptyRecycleBin $true
Powershell WhatIf syntax is supported. CompactAllVDI will show what will happen but not actually do anything when called with -WhatIf. Also, the standard Get-Help commands work with CompactAllVDI.

Final notes: We've tested this script on Windows 10 and Windows server 2019 machines and dozens of VMs. No problems with file corruption yet, but as always, backups are highly recommended. The script requires either version 3.0.2 of CloneVDI or version 4.0.1+.

The script is signed with our Authenticode cert. Modifications require either signing it with your own cert or stripping the existing certificate out for Powershell to run it. Powershell's execution policy may need to be changed on your system. I recommend AllSigned (Set-ExecutionPolicy AllSigned) to require a signature on all Powershell scripts if you have your own trusted cert, or RemoteSigned to require all downloaded scripts be signed. Setting a policy of Bypass opens your system to more malicious activity than I am comfortable with.
-------------------------------------------

All code is now available on GitHub: https://github.com/ethan8989/CompactAllVDI

-------------------------------------------
[ Edit: 18-Mar-2020: Version 1.2.2 Reports disk space reduction for each vdi disk compaction and overall. ]
[ Edit: 11-Mar-2020: Version 1.2. Terminates orphaned VBoxSVC processes still running after all running VMs are shut down and VirtualBox closed. ]
Attachments
CompactAllVDI.zip
VDI compaction Powershell script
(12.57 KiB) Downloaded 74 times
Last edited by EthanH on 19. Mar 2020, 03:11, edited 2 times in total.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Thanks for reporting back, and for contributing the script.
Lisa Miller
Posts: 17
Joined: 11. Mar 2020, 02:30

Re: CloneVDI tool - Discussion & Support

Post by Lisa Miller »

Hello,

I'm not sure if it is possible with CloneVDI to select a partition to create a RAW image or a VDI file from Windows 7 pro 64. At least I can't figure out how to select a partition. I only see how to select a single file.

Greeting
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

CloneVDI isn't a partition manager, it's a virtual disk cloning and tuning tool. What fundamentally are you trying to do?
 Edit:  From your other posts I see you are trying to P2V a physical Win7 system. For that you should use Disk2VHD as Scott has already recommended. The latter is simple to use, however the VHD format itself is pretty awful (prone to corruption), so once you have the VHD you'd run it through CloneVDI to convert it to VDI. You'd then build a VM around the VDI, trying to make the VM "recipe" as close as possible to your old Win7.

If the VDI contains unwanted partitions then you would deal with that after you have a working VM, by just deleting the unwanted partitions in Win7 itself. 
xonic
Posts: 3
Joined: 28. Mar 2010, 16:25
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Windows 10, Manjaro, Windows 2008

Re: CloneVDI tool - Discussion & Support

Post by xonic »

Hello,

With VirtualBox 6.1.4,

A VM with 2 snapshots:
1st: {a8a2b01f-2788-46aa-baec-1a232962321f}.vdi
2nd: {2d676aa3-9024-4d9c-b49d-31422205dd16}.vdi

Wanting to compact the 1st snapshot, I used CloneVDI with the Keep UUID and Compact options.
But the UUID is however modified.
The creation UUID is replaced by the parent UUID and the parent UUID is deleted.
As a result, there is no more connection with the parent and the VM can't find its snapshots anymore.

See the screenshots below
list of snapshots
list of snapshots
dir.PNG (14.85 KiB) Viewed 20772 times
original VDI
original VDI
vdi_ori.PNG (28.94 KiB) Viewed 20772 times
Clone VDI
Clone VDI
vdi_clone.PNG (26.73 KiB) Viewed 20772 times
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

This is not how CloneVDI is used. You can't compact a snapshot on its own. The fragmentation of the filesystem makes that impossible. You can merge and clone a snapshot chain, and while doing so you can compact the clone. That is what I'm seeing here. If you "Keep UUID" then you keep the UUID of the base VDI, which is what the guest OS sees - the snapshot's UUID is not relevent.

I don't think I would use "Keep UUID" on a snapshot chain anyway, unless you intend to delete the original VM with all snapshots.
xonic
Posts: 3
Joined: 28. Mar 2010, 16:25
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Windows 10, Manjaro, Windows 2008

Re: CloneVDI tool - Discussion & Support

Post by xonic »

In your readme notes file:
Q. Can it clone VDIs with snapshots?
A. Yes, CloneVDI v2.00 now has the ability to clone a snapshot VDI (the .VDI files with the strange
names that live in your \<VM>\Snapshots folder).

and also the paragraph

Cloning Snapshots
-----------------

can you explain to me how to compact a snapshot
Thanks you
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

It says it can clone VDIs which use snapshots, it does not say that it can compact individual snapshots.

I don't intend to explain this in more detail than is already in the release notes. CloneVDI is intended as a tool for experienced users, if you don't have enough technical understanding then perhaps you should use a different tool.
aalguqha
Posts: 5
Joined: 12. May 2020, 14:45

Re: CloneVDI tool - Discussion & Support

Post by aalguqha »

Hello,

I am a newbie to VirtualBox. While I am working on my VM, I usually take a snapshot after each major change. I am trying to restore my latest snapshot as my new VM. I have all the snapshots.vdi as well as the base.vdi, but when I start a new machine, the VirtualBox does not show the snapshots in the GUI. When I powered on the VM, I can only see my base VM. Is there a way to add the snapshots to the base.vdi, so I can then export the VM with the latest snapshot? My VM is Linux Mint 19.2, this is based on Ubuntu 18.04. My second question, can I use the cloneVDI to create my latest snapshot as a stand a lone VM? FYI, I copied my VirtualBox VM folder into external hard drive. So, the copy that I have is a basic copy (ctrl+c). MY VM is about 1.08 TB, does CloneVDI has a limitation of the .VDI size? I highly appreciate anyone who can help me to restore my work. Thanks in advance for all your help :)
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

Yes, you can create a merged VDI from the snapshots. See the release notes PDF in the root page: you need to read the (short) section on cloning snapshots very carefully.
aalguqha
Posts: 5
Joined: 12. May 2020, 14:45

Re: CloneVDI tool - Discussion & Support

Post by aalguqha »

mpack wrote:Yes, you can create a merged VDI from the snapshots. See the release notes PDF in the root page: you need to read the (short) section on cloning snapshots very carefully.
Great, I will try it tonight. Is there a way to use CloneVDI in Linux? If so, can you direct me to a discussion to follow the steps. I could not see anyone using the tool in Linux. If not, I guess I have to copy my folder into a hard disk and try it with Windows OS. Thanks a gain for your help.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CloneVDI tool - Discussion & Support

Post by mpack »

CloneVDI reportedly runs well under Wine on a Linux or MacOS host. The steps are trivial and need no discussion beyond what's in the release notes.
aalguqha
Posts: 5
Joined: 12. May 2020, 14:45

Re: CloneVDI tool - Discussion & Support

Post by aalguqha »

mpack wrote: Well, if you don't have a backup of all the important files then the old VM is destroyed.
The following may help you to recover your data:
Copy all of the VDI files (all snapshots plus the base VDI) into a single folder. Use the backup copies - your previous repair attempts probably damaged the copies currently on your hd.
Identify the latest snapshot VDI and clone it using CloneVDI.
Build a new VM around the clone - you cannot install the clone VDI in the original VM.
Note that Windows guests will quite likely need to be reactivated.
One thing I forgot to mention: the clone will be a stand alone VDI, so no snapshots required. Once the new VM is created and tested you can delete all the files associated with the old VM.
I have successfully clone my snapshots.vdi to a single stand a lone VDI. In the above quotes, you have answered this in Sepember 29, 2014. I am little bit confused of what are the requirements to build a new VM that is based on the clone_snapshots.vdi. Do we need to have the entire VM folder (this folder has base.vdi + all_snapshots.vdi + clone_snapshots.vdi) to build the new VM? or the clone_snapshots.vdi alone is enough to create the new VM? I hope that you can answer this question. Thanks again for your help.
Post Reply