Encrypting VM expands to maximum disk capacity - new?

[Zapados]

Host: Windows 10
Guest: Windows 10

I have a 128 GB dynamic disk which has about 50 GB used. When the VM is unencrypted I can shrink this down to be ~50 GB. But when I encrypt the VM, it takes up 128 GB.

I have another VM which I did the same thing to, and it's encrypted but the size is only 68.3 GB, with a 128 GB dynamic disk. What did I do there that I'm missing on this new VM?

Did something change/break in a newer version of VirtualBox?

Thank you for your help!

[mpack]

Are you referring to VirtualBox's own encryption? The following assumes: yes.

I would not expect (1) enabling encryption on its own to fill the disks. But, if you followed it up with (2) something that writes to all areas of the disks then yes, that's what I'd expect. E.g. even writing all zeroes would result in encrypted/randomized data filling all the sectors, which will maximize the VDI.

As to what makes the second VM different, you obviously omitted step (2).

I don't quite know what you mean by "when I shrink this down". But regardless of whether you mean compaction or compression, data files filled with random looking garbage will not compact or compress well. This is inherent in the nature of what encryption is.

[Zapados]

Sorry I did not have enough details in my first post.

For the disk which is maxing out at 128 GB here is what I did:

• 1. Unencrypted the disk in VirtualBox.
  2. Followed this procedure (How-to Geek "How to Shrink a VirtualBox Virtual Machine...") to shrink it (defragging on guest, then using SDelete to "zero" all free space on guest, then using VBoxManage to compact the disk on host).
  3. Disk size (size of .vdi) is now much smaller than 128 GB, ~50 GB.
  4. Enabled encryption in VirtualBox.
  5. Disk size (size of .vdi) balloons back up to 126 GB!

On the VM which doesn't have this problem I believe I followed a similar path but it's an older VM and I'm not sure exactly how I got there.

I don't see how those steps are causing the .vdi to blow back up to max capacity when I enable encryption. Any thoughts?

[fth0]

Sorry I did not have enough details in my first post.

And also in your second post.

mpack had to guess about what encryption you're talking about, and it still seems not clear to me either. Here's why it's important:

I've just tested that the VirtualBox Disk Encryption doesn't enlarge the disk size in general, so if you're talking about the VirtualBox Disk Encryption, we have something to investigate further. Alternative full disk encryption technologies like VeraCrypt or LUKS deliberately encrypt all sectors of a disk, so that the attacker cannot discover which sectors are in use and which are not, so if you're talking about one of those full disk encryptions, the behavior is expected.

FWIW, I'm using full disk encryption on the host, and neither the VirtualBox Disk Encryption nor any full disk encryption in the guest.

[Zapados]

@mpack was correct in his guess. The only encryption I'm playing with is the one that's part of VirtualBox, where you need a password to boot/access the VM.

I'll edit my prior post.

So what's strange is I was able to get the encryption to work without expanding the disk, but it took a LOT of effort. I followed the steps located here: www[dot]virtualbox[dot]org/ticket/14977 posted by Maxxim, but had to tweak some of his commands...ntfsclone --overwrite /dev/sdb2 /dev/sda2 for example.

Then when I encrypted the disk, it stayed a small size. I don't think this should have mattered. Is there some weird case where some data stays at the end of the drive and then the encryption gets everything in between and you get a large size? That doesn't make much sense right?

[fth0]

Well, I think I can explain the behavior now:

The VDI format distinguishes two types of absent blocks: FREE (-1) blocks are blocks that haven't been allocated yet, and ZERO (-2) blocks are blocks that contain only zeroes.

Since VirtualBox generally doesn't know which file system is used by the guest OS inside the virtual hard disk image, the VBoxManage modify medium --compact command simply identifies blocks containing only zeroes and marks them as ZERO before removing them. Note that the content of these blocks isn't necessarily free space, but can also be part of existing files. For the same reason, the VirtualBox Disk Encryption probably recreates the blocks with encrypted zeroes, so that an attacker cannot identify blocks of zeroes in existing files.

Bonus question for the author of CloneVDI: Since CloneVDI knows (some of) the file systems, does it mark the unused blocks it removes as FREE or as ZERO?

[mpack]

Since CloneVDI knows (some of) the file systems, does it mark the unused blocks it removes as FREE or as ZERO?

When compacting you mean? It actually does neither, because it does the compaction in the course of cloning the VDI, it's not marking an existing VDI. Any blocks it doesn't need are never allocated, which is the same as "free". It only marks blocks as zero if they need to exist (i.e. compaction doesn't remove them) and are filled with zeros.

However, CloneVDI can't do anything to the filesystem of an encrypted VDI since it has no access.

[fth0]

Since CloneVDI knows (some of) the file systems, does it mark the unused blocks it removes as FREE or as ZERO?

When compacting you mean? It actually does neither, [...]

Yes, and although my question apparently wasn't worded precisely enough ("mark" was meant to refer to the block map entries in the resulting VDI), you've effectively answered it.

To sum it up:

1. When compacting, CloneVDI can mark VDI blocks as FREE in the VDI block map of the resulting VDI, when it has knowledge about the file system used inside the VDI.
2. VBoxManage modify medium --compact can only mark VDI blocks as ZERO in the VDI block map of the resulting VDI, because it has no knowledge about the file system used inside the VDI.
3. When the VirtualBox Disk Encryption is activated, it recreates VDI blocks marked as ZERO in the VDI block map and fills them with encrypted zeroes.

[mpack]

o When the VirtualBox Disk Encryption is activated, it recreates VDI blocks marked as ZERO in the VDI block map and fills them with encrypted zeroes.

I'd be surprised if that is how it works.

I admit I have not checked out the encryption part of source code, due to lack of interest and extreme laziness; but if it was me then encryption would be done as a new wrapper around the VDI layer, responsible for encrypting and decrypting blocks as they pass in and out of the VDI. That would mean that the VDI layer never marks blocks as zero on write because it never sees a zero block. The only time it will ever see a zero block on read is if the VDI already existed and an OS was installed and working before encryption was turned on.

[fth0]

I did the following experiment to verify my statement (and to surprise you ):

I created a VM with an empty 1 GB VDI, verified that the VDI block map was filled with FREE (-1), booted the VM from a Linux Mint ISO (Live System), opened a terminal, became root (sudo bash), and proceeded as follows:

1. When filling the virtual hard disk with zeroes (cat /dev/zero > /dev/sda) for the first time, the VDI did not change: the VDI block map was still filled with FREE (-1), and no VDI blocks were added.
2. When filling the virtual hard disk with random data (cat /dev/urandom > /dev/sda), VDI blocks were added and the VDI block map was updated accordingly.
3. When filling the virtual hard disk with zeroes (cat /dev/zero > /dev/sda) for a second time, the VDI blocks were filled with zeroes and the VDI block map kept referencing them.
4. When shutting down the VM and executing VBoxManage modifymedium --compact, the VDI blocks were removed and the VDI block map was filled with ZERO (-2).
5. When encrypting the VM (from within the VirtualBox Manager), new VDI blocks were added with random looking data, and the VDI block map referenced them again.
6. When decrypting the VM (from within the VirtualBox Manager), the VDI blocks were removed and the VDI block map was filled with ZERO (-2) again.

Now I'm confident that I have not fully understood the FREE and ZERO flags (yet).

[tinmanjk]

So what's strange is I was able to get the encryption to work without expanding the disk, but it took a LOT of effort. I followed the steps located here: www[dot]virtualbox[dot]org/ticket/14977 posted by Maxxim, but had to tweak some of his commands...ntfsclone --overwrite /dev/sdb2 /dev/sda2 for example.

The steps by Maxxim from the ticket were:
1. start VM, clean disk, defragment disk, zero free space, shutdown VM
2. add new VDI of equal size to VM
3. edit .vbox file of VM to add encryption to new VDI (refer to old VDI for format; could not find any way to do this via GUI)
4. add live CD to VM (e.g. SystemRescueCD), start VM, boot into live CD
5. transfer partition table + bootloader to new disk (e.g. 'dd if=/dev/sda of=/dev/sdb bs=512 count=63')
6. transfer partition data using ntfsclone (e.g. 'ntfsclone --overwrite /dev/sdb1 /dev/sda1'; repeat for other partitions)
7. shutdown VM, remove old VDI

To save people from going to this ordeal, I want to expand on that a bit with what worked for me with Windows10 guest - there is no going around the need for a new vdi. Instead of cloning via Live disk which is difficult, I tried the buillt-in clonemedium command and it worked.

The steps that worked for me building on that were:
1. shutdown vm - (merge all snapshots so that you don't have any snapshots lefts and only the base disk)
2. decrypt vm (uuid of the disk for example 20979a0b-21df-4696-bb7a-e104639b5df1vboxmanage list hdds to get)
$ vboxmanage encryptmedium 20979a0b-21df-4696-bb7a-e104639b5df1 --oldpassword -
3. start VM, zero free space via sdelete -z C:, sdelete -z D: etc, shutdown
4. compact medium (actual shrinking)
$ vboxmanage modifymedium --compact 20979a0b-21df-4696-bb7a-e104639b5df1
5. clone via built-in command into a new VDI
$ vboxmanage clonemedium 20979a0b-21df-4696-bb7a-e104639b5df1 NewVdiName.vdi
6. find the new UUID of the new disk look for NewVdiName.vdi (e.g. 3bd07f86-e80a-4c86-84e6-424699c009bd)
$ vboxmanage list hdds
7. replace vdi for the vm with the cloned one. This is a bit tricky, cause you have to look at your *vbox file for the vm where the configuration is saved and more specifically StorageControllers tag for info that is required by the storageattach command: name, port and device
<StorageControllers>
<StorageController name="SATA" type="AHCI" PortCount="4" useHostIOCache="false" Bootable="true" IDE0MasterEmulationPort="0" IDE0SlaveEmulationPort="1" IDE1MasterEmulationPort="2" IDE1SlaveEmulationPort="3">
<AttachedDevice type="HardDisk" hotpluggable="false" port="0" device="0">
<Image uuid="{20979a0b-21df-4696-bb7a-e104639b5df1}"/>
</AttachedDevice>
</StorageController>
</StorageControllers>
With this info "SATA", port and device both 0, we can replace the vdi with the new cloned one.
$ vboxmanage storageattach Windows10 --storagectl "SATA" --device 0 --port 0 --type hdd --medium 3bd07f86-e80a-4c86-84e6-424699c009bd
8. Now you can actually encrypt your vm again
$ vboxmanage encryptmedium 3bd07f86-e80a-4c86-84e6-424699c009bd --cipher AES-XTS256-PLAIN64 --newpasswordid Windows10 --newpassword -

Done.
Now you can test that everything works.
Then get rid of the old medium.
VBoxManage closemedium disk 20979a0b-21df-4696-bb7a-e104639b5df1 --delete
Comments are welcome how to streamlines this more.