How would I shrink a macOS guest VDI?

[IOOI]

Hi there,

I have a virtual macOS installation (10.12.6) which uses 36,8 GB of it's hard disk space (VDI is of type "dynamically growing" until 2TB) but the VDI now has a size of 74GB, since there was once more stuff (older Xcode betas) on the disk which I deleted.

Unfortunately deleting stuff inside a VDI does not shrink it in size. So I googled how to do this, but found only how tos for Windows or Linux guests.

How would I defragment the VDI and erase the free space (can this be done using disk utility inside the guest?)?

thanks in advance,

IOOI

[socratis]

So I googled how to do this, but found only how tos for Windows or Linux guests.

Avoid Google, use the User Manual .
Ch. 8.23. VBoxManage modifymedium:

For Mac OS X guests, use the "diskutil secureErase freespace 0 /" command line from an elevated Terminal.

I had to ping the manual maintainers a couple of times to change that, because Apple kept changing the way they're doing things, but I found that this command works from 10.4 to even the 10.13b.

[mpack]

Unfortunately deleting stuff inside a VDI does not shrink it in size.

Shrinking the partition (with a guest partition manager) will limit how large the VDI can grow, as it only grows when disk sectors are first used.

[SteveMacGeek]

I would like to warn readers of this particular thread NOT to attempt using the following, at least not on 10.14.x Mojave:

"diskutil secureErase freespace 0 /"

This hoarked my Mojave .vdi completely.

It may be that the issue is the volume is APFS, and using / is simply too much -- perhaps you have to specify something like /dev/disk1, or /dev/disk0s1.

In any case, take great care!

[socratis]

Do you have an alternative solution? I haven't installed 10.14 yet, so I haven't studied the ins and outs of APFS. Is this another case where Apple screwed up again? As usual?

[SteveMacGeek]

No, APFS is definitely NOT a screw-up. It's a major improvement on HFS+, in terms of performance, security, and reliability.

It is, however, optimized for SSDs. So typically, you would never ever need to do a secure erase of unused space, because the built-in encryption is plenty of security. But repeatedly writing zeros to an SSD would wear out the SSD, so I think that option is normally not available.

But you CAN do it with diskutil.

I think the idea of using the volume root "/" with diskutil secureErase is just too dangerous -- I think that might include all those special APFS component / partitions that are magically stitched together.

But without doing that, I don't know how the volume can be compacted.

Stephen

[socratis]

APFS was definitely a screw-up when introduced. Untested, unproven, bugs riddled.

It is, however, optimized for SSDs

Please tell that to my external hard drive, that it should upgrade itself to an SSD...

But you CAN do it with diskutil.

You actually didn't tell how...

I think the idea of using the volume root "/" with diskutil secureErase is just too dangerous

Then someone hasn't done their homework... hint: Apple.

BTW, I don't know how many times I have to say that: a 10.14/15 guest is NOT supported.

[SteveMacGeek]

APFS was definitely a screw-up when introduced. Untested, unproven, bugs riddled.

APFS was road-tested on iPhone and Apple watch for a couple years before being introduced on macOS. The biggest issue with the introduction of APFS on macOS was the forced conversion of Apple's hybrid drives (part SSD, part hard disk). In my own development I had to accommodate the changes in API due to Apple's changing / eliminating API calls that it had deprecated years earlier. So, like me, a lot of other software vendors may have been caught flat-footed.

You actually didn't tell how...

Sorry I wasn't clear. What I meant was, diskutil will allow you to execute a secure freespace command from the terminal. It's just...deadly as proposed in section 8.24:

Code: Select all

diskutil secureErase freespace 0 /

Then someone hasn't done their homework... hint: Apple.

OR -- maybe Section 8.24 should be revised to say "Don't try this on macOS 10.14 or greater"

BTW, I don't know how many times I have to say that: a 10.14/15 guest is NOT supported.

Do you not want forum members to explicitly not post questions for 10.14 Guests?

Stephen

[fth0]

IMO the example command in the VirtualBox User Manual is simply wrong ok*. and should be corrected. The example commands for the other OSs do not specify the (necessary) disk device, so the easiest change would be to simply remove the trailing '/', so that none of the commands work right out of the box.

*The commands all expect a device parameter. The file system root '/' could either be interpreted as no device or as all devices mounted somewhere in the file system hierarchy. The consequence would be either no operation taking place, or all devices being affected. I think both are usually not wanted. The device parameter can be given in form of a mount point, so the file system root '/' specifies the device mounted at this mount point.

[SteveMacGeek]

The file system root '/' could either be interpreted as no device or as all devices mounted somewhere in the file system hierarchy. The consequence would be either no operation taking place, or all devices being affected. I think both are usually not wanted.

@fth0 - agreed. So the question is, where is the main device for the user's boot volume on APFS?

Perhaps a safer way to zero out the sector data would be a more low-rent app that doesn't try to use the system hard disk utility to do its work. I recall one such app -- years ago -- that simply kept allocating a sectors in a huge file, then wrote out zeros to that file, then deleted the file. While that might not be enough for a "secure" erase on a non-ssd hard disk, it should do the job for he purposes of allowing VirtualBox to compact the filesystem space.

Stephen

[socratis]

a 10.14/15 guest is NOT supported.

OR -- maybe Section 8.24 should be revised to say "Don't try this on macOS 10.14 or greater"

Do you not want forum members to explicitly not post questions for 10.14 Guests?

IMO the example command in the VirtualBox User Manual is simply wrong* and should be corrected.

The "not supported" part means that the User Manual stays "as is". You can't have instructions on the User Manual for something that's not supported. If you find a proper workaround/solution, and 10.14/15 become officially supported, then the User Manual may be corrected to reflect that aspect.

How do I know? Because that's exactly what happened 7 years ago, and amended 3 years later, after Apple once again changed their tune: #11175: Add documentation for zeroing Mac OS X guests before using the 'modifyhd --compact' operation...

Be sure that I'm going to be the first that wakes that ticket again, once a proper solution is found. Since I haven't installed 10.14/15 yet, I can't personally do the testing. It's on my "to-do list" for the longest of times...

@SteveMacGeek
You already opened a new thread about this, Strategies for compressing huge .vdi for upload/download, which practically talks about the same thing. Can you focus on that thread and leave this one (that was resolved in 2017) sleep? You gave a warning for 10.14 guests, fine. But let's keep the discussion about the way to compress an OSX 10.14 VDI in your new, more relevant thread.

[granada29]

I have just manged to compact a Mojave boot drive using the following commands.

1. Run the VM and login as the admin user
2. Use Disk Utility to find the device ID of the volume you wish to compact - please note - you need the volume ID, not the whole disk
   2a. Alternatively use 'diskutil list' from the command line to find the device ID of the volume
3. In the Terminal, 'sudo su -' to give yourself elevated privileges.
4. diskutil secureErase freespace 0 "device ID from 2 - without quotes"
5. Wait until this completes - it may take some time depending on the type of disk (SSD vs HDD) and the size of the disk
6. Shutdown the VM cleanly - i.e. a complete shutdown and not "saved state" or "power off"
7. In the VirtualBox Manager, right click the VM entry in the left column and select 'Show in Finder'. This will open a Finder window showing your VM.
8. Open Terminal - do NOT elevate privileges
9. Type 'VBoxManage modifymedium --compact ' then drag the .vdi file from the Finder window to complete the command. Please note that there should be a space character between 'compact' and the beginning of the file name. The normal rules of escaping spaces in paths are done automatically if you use drag and drop. If you choose to type the path manually, please remember to take into account any spaces in the path.
10. Press 'return' and wait until the command completes.

That's it. It seems the documentation in Chapter 8 is misleading and it is much safer to specify the deviceID rather than the root path "/"

I kept a copy of the original disk before doing any of this - before/after comparisons below

Code: Select all

bryan@number9 BuildInkscape % VBoxManage modifymedium --compact ./BuildInkscape.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
bryan@number9 BuildInkscape % ls -l
total 136300576
-rw-------  1 bryan  staff         4299 28 Oct 08:49 BuildInkscape.vbox
-rw-------  1 bryan  staff         4299 25 Oct 10:35 BuildInkscape.vbox-prev
-rw-------  1 bryan  staff  26684162048 28 Oct 08:59 BuildInkscape.vdi
-rw-------@ 1 bryan  staff  43088084992 24 Oct 08:49 BuildInkscape.vdi.save.vdi
drwx------  6 bryan  staff          192 28 Oct 08:24 Logs
bryan@number9 BuildInkscape % 

Following this - I ran the VM (using the compacted disk) and it 'just worked'.

[socratis]

Use Disk Utility to find the device ID of the volume you wish to compact - please note - you need the volume ID, not the whole disk

That's it. It seems the documentation in Chapter 8 is misleading and it is much safer to specify the deviceID rather than the root path "/"

1. It's not misleading because ... it works with ALL supported OSX guests.
2. If this finding of yours works with previous guests, from 10.16^[1] 10.6 to 10.13, and it works for 10.14/15 as well, then it's something to definitely consider! It's going to make the instructions in the User Manual a tad more awkward, but who cares...

-rw-------  1 bryan  staff  26684162048 28 Oct 08:59 BuildInkscape.vdi
-rw-------@ 1 bryan  staff  43088084992 24 Oct 08:49 BuildInkscape.vdi.save.vdi

I take it that the ".vdi.save.vdi" is the original VDI. That's quite some savings there!

---

^[1]: Thanks to 'rpmurray' for spotting the typo!

[socratis]

You already opened a new thread about this, Strategies for compressing huge .vdi for upload/download, which practically talks about the same thing. Can you focus on that thread and leave this one (that was resolved in 2017) sleep? You gave a warning for 10.14 guests, fine. But let's keep the discussion about the way to compress an OSX 10.14 VDI in your new, more relevant thread.

I think I'm going to reverse that decision, since there's no actual back and forth in that other, newer thread. I'm going to lock the other thread, and redirect people here.

[SteveMacGeek]

@granada29 -- So -- to be clear -- in the following diskutil list, APFS Volume Prezdrive is my mounted boot volume. Its Device ID is disk1s1 -- what Disk Utility refers to as the "BSD device node", correct?

Code: Select all

$ diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         1.0 TB     disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +1.0 TB     disk1
                                 Physical Store disk0s2
   1:                APFS Volume Prezdrive               925.2 GB   disk1s1
   2:                APFS Volume Preboot                 42.8 MB    disk1s2
   3:                APFS Volume Recovery                510.4 MB   disk1s3
   4:                APFS Volume VM                      9.7 GB     disk1s4

Stephen