Can I compact VDI this way?

Discussions about using Linux guests in VirtualBox.
Post Reply
Random_Consultant
Posts: 18
Joined: 24. Aug 2015, 21:02

Can I compact VDI this way?

Post by Random_Consultant »

I recently made the rash error of increasing the size of one of my guest VDIs due to the 5.2.6 GA bug.

Long story short: I don't want the extra space anymore.

From what I've read online, folks have made use of a software known as Zerofree.

Is is not possible to simply shrink the main partition via GParted (leaving unallocated free space on the drive) and run vboxmanage modifymedium disk.vdi -compact?

I'm new to this procedure so I'd like some feedback (please). :D

TIA

- T.R.G.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Can I compact VDI this way?

Post by socratis »

First a question: why did you increase the size as a response to the 5.2.6 GAs issue?

Second, your thinking is almost correct, but I would do the following:
  1. Run the "zerofree" command using a live CD like "SystemRescueCd"
  2. Run the "VBoxManage modifymedium "<YourVDI> --compact" on your host.
  3. Resize/shrink the partition with GParted.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Random_Consultant
Posts: 18
Joined: 24. Aug 2015, 21:02

Re: Can I compact VDI this way?

Post by Random_Consultant »

To answer the question:
Running Linux Mint, I attempted to logon and was greeted with a message displaying possible causes for the error. One of them happened to be insufficient disk space. I purposely created the VDI to hold just the minimum amount of disk space necessary for my purposes (learning). The reason I thought of increasing the disk size was because I recently encountered an issue where I literally ran out of disk space for normal operation and was forced to use GParted to reallocate the newly added space on the drive. I clearly wasn't thinking correctly this time around as I knew that I hadn't taken up the entirety of the drive upon installation of the buggy 5.2.6 GA. For whatever reason, I panicked and decided to give the increase a go. After that had failed, I came on here and started looking to see if anyone had the same issue. Eventually, I was able to download the more recent GA and resume things normally.

I was really out of it yesterday... :lol:

Back to my question:
I do not yet have Zerofree installed on my guest machine. From what you've mentioned thus far, I can assume I don't need to as long as I use that particular live-cd?

May I ask why GParted wouldn't be run before shrinking the drive?
Last edited by socratis on 19. Jan 2018, 20:29, edited 1 time in total.
Reason: Removed unnecessary verbatim quote of the whole previous message.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can I compact VDI this way?

Post by mpack »

Random_Consultant wrote:May I ask why GParted wouldn't be run before shrinking the drive?
I see no "shrink drive" instruction, whether before or after GParted. In fact VirtualBox has no such feature, because of the obvious destructive potential.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Can I compact VDI this way?

Post by socratis »

Random_Consultant wrote:I do not yet have Zerofree installed on my guest machine. From what you've mentioned thus far, I can assume I don't need to as long as I use that particular live-cd?
You don't want 'zerofree' installed in your guest. In fact I'm not sure if 'zerofree' can work on a drive that it is located on and it is the boot drive. That's why I said run it from a live CD, because your guest's drive can be modified at will.
Random_Consultant wrote:May I ask why GParted wouldn't be run before shrinking the drive?
Just like mpack said, no drive shrinking, only partition "rearranging". And I'm not sure what you mean running X before X (does what it does). Doesn't compute...
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Random_Consultant
Posts: 18
Joined: 24. Aug 2015, 21:02

Re: Can I compact VDI this way?

Post by Random_Consultant »

mpack wrote:
Random_Consultant wrote:May I ask why GParted wouldn't be run before shrinking the drive?
I see no "shrink drive" instruction, whether before or after GParted. In fact VirtualBox has no such feature, because of the obvious destructive potential.
Good catch!

Also, thanks for clarifying. In my mind, that is what it was doing; that is why I asked what I did.

I wasn't sure how it was doing the "compacting".
Random_Consultant
Posts: 18
Joined: 24. Aug 2015, 21:02

Re: Can I compact VDI this way?

Post by Random_Consultant »

socratis wrote:
Random_Consultant wrote:May I ask why GParted wouldn't be run before shrinking the drive?
Just like mpack said, no drive shrinking, only partition "rearranging". And I'm not sure what you mean running X before X (does what it does). Doesn't compute...
I'm wondering why the VDI can be compacted before using GParted. Why is the recommended procedure compacting the VDI before running GParted? Is compacting not shrinking the VDI?

I ask because to me, that does not compute.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: Can I compact VDI this way?

Post by socratis »

Let me "sort of" draw the process. Warning: Oversimplification.

In the following ASCII-art representation of the state of the VDI:
  • the letters are occupied sectors,
  • the spaces " " are empty/unallocated sectors,
  • the dashes "-" are sectors that were allocated at some point, but not anymore,
  • the stars "*" are sectors that have been marked as "zeroed out" from zerofree/sdelete/other,
  • and finally, the "||" is where you divide your disk with GParted/DiskUtility to have a smaller accessible partition.
In the original VDI state, the VDI contains data (randomly located), free, and previously used sectors:
+-----------------------------------------------------------------+
| l d --- od--- skkjs    gsosgg   ds    dl  k  sl   ks f  po ss-- |
| j s     k l---j--sd  v   k sd j  s dl kj s d  sd g jk ls d  js  |
| d  l---s j  o  ;   s d j lk-s---j-sl---a-s al sd  sd hk l as  5 |
| has   d h ig----w  2 4       pu t g  df b j; af s j--as-l-as--b |
+-----------------------------------------------------------------+
After running 'zerofree/sdelete/other' all the previously used sectors are marked as zero sectors:
+-----------------------------------------------------------------+
| l d *** od*** skkjs    gsosgg   ds    dl  k  sl   ks f  po ss** |
| j s     k l***j**sd  v   k sd j  s dl kj s d  sd g jk ls d  js  |
| d  l***s j  o  ;   s d j lk*s***j*sl***a*s al sd  sd hk l as  5 |
| has   d h ig****w  2 4       pu t g  df b j; af s j**as*l*as**b |
+-----------------------------------------------------------------+
After running 'VBoxManage modifymedium "<VDI>" --compact' all the zero sectors are emptied:
+-----------------------------------------------------------------+
| l d     od    skkjs    gsosgg   ds    dl  k  sl   ks f  po ss   |
| j s     k l   j  sd  v   k sd j  s dl kj s d  sd g jk ls d  js  |
| d  l   s j  o  ;   s d j lk s   j sl   a s al sd  sd hk l as  5 |
| has   d h ig    w  2 4       pu t g  df b j; af s j  as l as  b |
+-----------------------------------------------------------------+
After shrinking the partition with GParted/DiskUtility all the data are moved to the front, and they are restricted onto their own, smaller partition, and you don't allow anything to access the "No Access" partition:
+---------------------------------------++------------------------+
| ldodskkjsgsosggdsdlkslksfposs         ||                        |
| jskljsdvksdjsdlkjsdsdgjklsdjs         ||   N O    A C C E S S   |
| dlsjo;sdjlksjslasalsdsdhklas5         ||                        |
| hasdhigw24putgdfbj;afsjaslasb         ||                        |
+---------------------------------------++------------------------+
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can I compact VDI this way?

Post by mpack »

Q. What does zerofree do?
A. It can be used to zero-fill unused parts of a selected filesystem.

Q. What does VBoxManage compact do?
A. It deallocated zero blocks from the VDI (1MB blocks filled with zeroes).

Q. What does GParted do?
A. It changes what fraction of the disk is covered by a filesystem.

Note that GParted doesn't shrink or erase anything, it just writes a new data structure on the drive.

So now :-
Q. What would happen if I shrank a partition using GParted before running zerofree?
A. Then the filesystem would cover less of the disk when you ran zerofree, the remainder would be unreachable to zerofree, and those high blocks would hence not be zeroed and not be removed by VBoxManage compact.

p.s. The recommended procedure is not ideal either. Running gparted after zerofree causes data to be written to some of the blocks you just zeroed, lowering the effectiveness of the compaction step. It's a limitation of the official approach. The unofficial approach is to use CloneVDI, which doesn't require that you run zerofree, and doesn't care at which point you run GParted, because it discards unused blocks from the whole drive, not zero blocks from one part of it. CloneVDI will also be very much faster.
arQon
Posts: 228
Joined: 1. Jan 2017, 09:16
Primary OS: MS Windows 7
VBox Version: PUEL
Guest OSses: Ubuntu 16.04 x64, W7

Re: Can I compact VDI this way?

Post by arQon »

socratis wrote:You don't want 'zerofree' installed in your guest. In fact I'm not sure if 'zerofree' can work on a drive that it is located on and it is the boot drive.
It can, but it requires the system to be in single-user mode (ie init 1) and the drive to be mounted ro. (viewtopic.php?f=3&t=82623 has the details). As you say though, it's a lot easier to just use a live CD for stuff like this, especially if you're going to use tools like GParted as well anyway.

(and as mpack says, CloneVDI is the best option of all, provided you can run it).
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can I compact VDI this way?

Post by mpack »

arQon wrote:CloneVDI is the best option of all, provided you can run it).
Pretty much everyone can, as it runs natively on Windows hosts, and is just as happy under Wine on Linux and OS X hosts.
Random_Consultant
Posts: 18
Joined: 24. Aug 2015, 21:02

Re: Can I compact VDI this way?

Post by Random_Consultant »

@mpack @socratis

Thanks guys! :)
grahamperrin
Posts: 57
Joined: 27. Apr 2008, 15:19

Preparing to compact: FreeBSD-CURRENT host

Post by grahamperrin »

With FreeBSD-CURRENT, I seem to gain free space through preparing to compact virtual disk images.

Today, for example:
  • I recall having 160-something G free before booting systemrescuecd-amd64-6.1.2.iso and running zerofree
  • I have 180G free after the run of zerofree.
Manjaro.vdi actual size 63.58 GB when I began running VBoxManage … I'll revisit this post after the run completes.

Code: Select all

grahamperrin@momh167-gjp4-8570p:~ % date ; uname -v
Sun 29 Mar 2020 19:01:16 BST
FreeBSD 13.0-CURRENT #51 r357746: Tue Feb 11 03:54:19 GMT 2020     root@momh167-gjp4-8570p:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG 
grahamperrin@momh167-gjp4-8570p:~ % pkg query '%o %v %R' virtualbox-ose virtualbox-ose-kmod
emulators/virtualbox-ose 5.2.34_1 FreeBSD
emulators/virtualbox-ose-kmod 5.2.34 poudriere
grahamperrin@momh167-gjp4-8570p:~ % zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
copperbowl   448G   268G   180G        -         -    55%    59%  1.00x  ONLINE  -
grahamperrin@momh167-gjp4-8570p:~ % time VBoxManage modifymedium --compact 1e3306a9-3506-4bcc-9f98-d1cece4293d3
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
8.205u 4.794s 35:06.61 0.6%     936+176k 24+0io 10pf+0w
grahamperrin@momh167-gjp4-8570p:~ % zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
copperbowl   448G   268G   180G        -         -    55%    59%  1.00x  ONLINE  -
grahamperrin@momh167-gjp4-8570p:~ % 
… .vdi reduced from 63.58 GB to 39.73 GB but no change in the amount of free space.

Maybe a peculiarity of the host system using ZFS.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: Can I compact VDI this way?

Post by mpack »

Neither zerofree nor compaction gives you more free space.

"zerofree" literally means to zero already free sectors, so how could that add free sectors?

Compaction is a host operation, the guest shouldn't be affected at all (not unless it was "hiding" data in unused parts of the drive).
Post Reply