How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

This is for discussing general topics about how to use VirtualBox.
Post Reply
adrelanos
Posts: 22
Joined: 9. Sep 2018, 09:48

How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by adrelanos »

How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

What's the purpose? VMDKs in exported VirtualBox OVAs are compressed, yes, but better compression is possible by using tar with maximum gzip compression. However, a .tar.gz file that contains an .ova is harder to use and takes longer to import (requires untar + import). Therefore I would like to have an .ova file that contains a gzip compressed disk image .vmdk. This would have the advantage that double clicking an .ova file is still associated with VirtualBox. But if it's a .tar.gz that contains an .ova, that won't work.

If this is impossible using VirtualBox GUI as well as vboxmanage command line, how could one accomplish the .ova using gzip compression for its .vmdk images?

What I am not looking for is vboxmanage modifymedium --compact / zerofree because that is already discussed elsewhere, it's great, and I am already using that. Additional gzip compression on top of it would be even better.

Gzip compression of VMDK files seems to be supported in both the OVA standard as well as in VirtualBox source code.

Quote OVA standard:
Each file referenced by a File element may be compressed using gzip (see RFC1952). When a File
element is compressed using gzip, the ovf:compression attribute shall be set to “gzip”.
VMware has a bug OVF Package with compressed disks is currently not supported for OVF import (53260). It may seem besides the point but this further indicates that this is possible in theory.

VirtualBox source file tarvfs.cpp mentions gzip several times. So does ApplianceImplImport.cpp and ApplianceImplExport.cpp. Also VirtualBox IAppliance Interface Reference mentions gzip.

VirtualBox source code folder contains a file src/VBox/ValidationKit/tests/api/tdAppliance1-t5.ova which is I guess used as a unit test and that file describes gzip compression.

Code: Select all

    <File ovf:compression="gzip" ovf:href="tdAppliance1-t5-disk1.vmdk.gz" ovf:id="file1" ovf:size="509"/>
That further indicates that VirtualBox supports this. The only thing that I couldn't find out after hours of internet and source code research is how to create an OVA that uses gzip compression for the disk images it includes. One could:

1. export to .ovf instead,
2. manually or with a script compress the .vmdk with gzip,
3. then edit the .ovf
4. create a tar archive, the actual .ova

That might work? But seems rather cumbersome. Is there an existing tool which can help here?
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by scottgus1 »

One problem you'll have is that if you deviate from the OVF standard in making the OVA, then you won't be making an OVA anymore. You'll be making an 'ADRELANOS', that is, a private distribution method not compatible with OVF.

These options are available with the "vboxmanage export" command:
export <machines> --output|-o <name>.<ovf/ova/tar.gz>
[--legacy09|--ovf09|--ovf10|--ovf20|--opc10]
[--manifest]
[--options manifest|iso|nomacs|nomacsbutnat]
[--vsys <number of virtual system>]
[--vmname <name>]
[--product <product name>]
[--producturl <product url>]
[--vendor <vendor name>]
[--vendorurl <vendor url>]
[--version <version info>]
[--description <description info>]
[--eula <license text>]
[--eulafile <filename>]
Sections 1.14.3 and 8.11 cover the meaning of some of the options. There is a tar.gz option, so maybe what you want is already there. Or this may tar the whole OVA file, I don't know.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by fth0 »

AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format. The easiest way to check that is to search for "ovf:compression", which is neither written nor read nor interpreted anywhere in the VirtualBox 6.1.38 sources. You can also search for "ovf:size" to find the places where the compression option would have to be dealt with. In consequence, you'd have to implement the use of gzip compression yourself if you really wanted it.

What you've found are usages of the .tar.gz format, for example for the OCI Appliance import/export.
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by mpack »

fth0 wrote:AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format.
Are you sure? Or am I misinterpreting your statement? I know for a fact that VirtualBox supports stream compressed VMDK on import. In fact it's not restricted to import, it's a part of the normal VMDK handling: you can build a VM around a stream compressed VMDK and it will work, but VirtualBox will use a difference image to handle writes.

What I don't know is whether VirtualBox uses compression on the OVA files it creates. It includes the zlib library so I'd be surprised if not.
 Edit:  I checked the vmdk resulting from an exported appliance (OVA). The VMDK was stream compressed according to CloneVDI.

pedantic note: gzip is a piece of software, not a stream format. AFAIK the stream format is covered by RFC 1951 and is used by several tools, e.g. PNG. I imagine it also includes gzip. 
adrelanos
Posts: 22
Joined: 9. Sep 2018, 09:48

How to create a GZIP compressed OVA file?

Post by adrelanos »

My main question:
How to create a GZIP compressed OVA file?


Not "create OVA first, then wrap it into a TAR.GZ." please. A "directly" gzip compressed OVA.

An OVA file is supposedly just a TAR file and in theory should support GZIP compression. (See below for references.)

Create OVA from Command Line using TAR - GZIP - Broken

Trying to import a VirtualBox OVA that I created manually using the command line, I am getting a VERR_TAR_BAD_CHKSUM_FIELD error.

This was my command to manually create the OVA from command line using tar (broken):

Code: Select all

tar --gzip --create --verbose --directory=/home/user/temp --file temp.ova temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf
temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf
Trying to import... No... Failing...

Code: Select all

vboxmanage import temp.ova ; echo $?
Progress state: VBOX_E_IPRT_ERROR
VBoxManage: error: Appliance read failed
VBoxManage: error: Error reading OVA '/home/user/temp/temp.ova' (VERR_TAR_BAD_CHKSUM_FIELD)
VBoxManage: error: Details: code VBOX_E_IPRT_ERROR (0x80bb0005), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg*)" at line 471 of file VBoxManageAppliance.cpp
Create OVA from Command Line using TAR - Functional - But lacking GZIP Compression

If I drop --gzip from tar parameters it's functional.

This was my command to manually create the OVA from command line using tar (functional):

Code: Select all

tar --create --verbose --directory=/home/user/temp --file temp.ova temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf
Then import works.

Code: Select all

vboxmanage import temp.ova ; echo $?
[...]
0
Notes:

For the error VERR_TAR_BAD_CHKSUM_FIELD I have found the related VirtualBox source code. [1] [2]

What tar checksum does VirtualBox expect? I tried to learn more about if tar checksums exist and how they work but I couldn't figure out anything yet. See also Does the tar format support checksums?

Also tar parameter --format=ustar did not help.
  • What I am not looking for is vboxmanage modifymedium --compact / zerofree because that is already discussed elsewhere, it's great, and I am already using that. Additional gzip compression on top of it would be even better.
  • What my main goal? A gzip compressed `.ova` VirtualBox appliances that can be imported directly into VirtualBox without an additional `.tar.gz`.
  • Aren't VirtualBox ova's compressed anyhow? No. These are just tar files. Not gzip or xz compressed archives.
  • But the `.vmdk` files in `.ova` files are compressed? Yes, but the compression is limited. By using an additional `.tar.gz` with maximum compression, the `.ova` gets smaller.
  • Why not use an additional `.tar.gz` file? Compression, speed, usability, simplicity.
  • Why do you think that VirtualBox can import `.ova`s that are actually `.tar.gz` files? In How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export? I posted various references to gzip being mentioned in VirtualBox source code and OVA specification.
  • Is How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export? a duplicate? No. That one is about gzip compression of disk images inside OFV / OVA. This forum thread is about gzip compressed OVA. It's not exactly the same. It's a different approach that might accomplish the same end goal of better OVA compression.
  • Why am I mentioning all of this? Because most questions that I've seen go into direction of `vboxmanage modifymedium --compact` / `zerofree`. Therefore I wanted to be very specific here: An `.ova` file that is gzip compressed.
[1]

Code: Select all

/**
 * Validates the TAR header.
 *
 * @returns VINF_SUCCESS if valid, VERR_TAR_ZERO_HEADER if all zeros, and
 *          the appropriate VERR_TAR_XXX otherwise.
 * @param   pTar                The TAR header.
 * @param   penmType            Where to return the type of header on success.
 */
static int rtZipTarHdrValidate(PCRTZIPTARHDR pTar, PRTZIPTARTYPE penmType)
{
    /*
     * Calc the checksum first since this enables us to detect zero headers.
     */
    int32_t i32ChkSum;
    int32_t i32ChkSumSignedAlt;
    if (rtZipTarCalcChkSum(pTar, &i32ChkSum, &i32ChkSumSignedAlt))
        return VERR_TAR_ZERO_HEADER;

    /*
     * Read the checksum field and match the checksums.
     */
    int64_t i64HdrChkSum;
    int rc = rtZipTarHdrFieldToNum(pTar->Common.chksum, sizeof(pTar->Common.chksum), true /*fOctalOnly*/, &i64HdrChkSum);
    if (RT_FAILURE(rc))
        return VERR_TAR_BAD_CHKSUM_FIELD;
    if (   i32ChkSum          != i64HdrChkSum
        && i32ChkSumSignedAlt != i64HdrChkSum) /** @todo test this */
        return VERR_TAR_CHKSUM_MISMATCH;
[2]

Code: Select all

/*
 * Copyright (C) 2010-2020 Oracle Corporation
 *
 * This file is part of VirtualBox Open Source Edition (OSE), as
 * available from http://www.virtualbox.org. This file is free software;
 * you can redistribute it and/or modify it under the terms of the GNU
 * General Public License (GPL) as published by the Free Software
 * Foundation, in version 2 as it comes in the "COPYING" file of the
 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
 *
 * The contents of this file may alternatively be used under the terms
 * of the Common Development and Distribution License Version 1.0
 * (CDDL) only, as it comes in the "COPYING.CDDL" file of the
 * VirtualBox OSE distribution, in which case the provisions of the
 * CDDL are applicable instead of those of the GPL.
 *
 * You may elect to license modified versions of this file under the
 * terms and conditions of either the GPL or the CDDL or both.
 */
Create OVA from Command Line using VBoxManage - GZIP - Broken

This is might actually be a different approach from above. The following might create (if that was working) an ova that is wrapped into a separate tar.gz?

Syntax shown when running "VBoxManage export" (built-in help).

Code: Select all

VBoxManage export           <machines> --output|-o <name>.<ovf/ova/tar.gz>
functional:

Code: Select all

vboxmanage export test --output test.ova
So a uncompressed ova can be created using vboxmanage.

broken:

Code: Select all

vboxmanage export test --output test.tar.gz

Code: Select all

vboxmanage export test --output test.ova.tar.gz
In both cases...
VBoxManage: error: Appliance file must have .ovf or .ova extension
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance, callee nsISupports
VBoxManage: error: Context: "Write(Bstr(strOvfFormat).raw(), ComSafeArrayAsInParam(options), Bstr(pszAbsFilePath).raw(), progress.asOutParam())" at line 1934 of file VBoxManageAppliance.cpp
Is this a bug or am I using it wrong?
adrelanos
Posts: 22
Joined: 9. Sep 2018, 09:48

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by adrelanos »

scottgus1 wrote:One problem you'll have is that if you deviate from the OVF standard in making the OVA, then you won't be making an OVA anymore.
I think this is compatible with the OVF standard. Here's what I found out:

1. I went to https://en.wikipedia.org/wiki/Open_Virt ... ion_Format to learn more about OVF
2. it links to the specification https://www.dmtf.org/standards/ovf
3. DSP0243 2.1.1 Open Virtualization Format Specification 27 Aug 2015 Standard View
Interesting. Click.
4. https://www.dmtf.org/sites/default/file ... _2.1.1.pdf
5. Title of pdf is "Open Virtualization Format Specification", good.
6. Search of "gzip" inside the pdf.
7. Found this:
Each file referenced by a File element may be compressed using gzip (see RFC1952). When a File
536 element is compressed using gzip, the ovf:compression attribute shall be set to “gzip”.
So in conclusion, OVF standard supports it.

VirtualBox seems to support to import something along gzip too. See references above. Just creation seems to be undocumented.
scottgus1 wrote: These options are available with the "vboxmanage export" command:
export <machines> --output|-o <name>.<ovf/ova/tar.gz>
[--legacy09|--ovf09|--ovf10|--ovf20|--opc10]
[--manifest]
[--options manifest|iso|nomacs|nomacsbutnat]
[--vsys <number of virtual system>]
[--vmname <name>]
[--product <product name>]
[--producturl <product url>]
[--vendor <vendor name>]
[--vendorurl <vendor url>]
[--version <version info>]
[--description <description info>]
[--eula <license text>]
[--eulafile <filename>]
Sections 1.14.3 and 8.11 cover the meaning of some of the options. There is a tar.gz option, so maybe what you want is already there. Or this may tar the whole OVA file, I don't know.
That's actually a different approach which I've created a different forum thread for. Hope that's OK. I think it's sufficiently different to warrant a different forum thread. The end goal of better compression while still having OVA files that are easily importable into VirtualBox (double click on OVA) is the same. But the technical approach is a different one. This approach in this forum thread uses gzip on the VMDK which is then to be defined in the OVF ("inner layer"). The other approach in a separate forum thread uses gzip for the OVA ("outer layer").

- This forum thread: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

- Alternative approach: {dead link removed}
Last edited by mpack on 10. Sep 2022, 16:01, edited 1 time in total.
Reason: Dead link
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by mpack »

Cross posting is against the forum rules, and I don't believe the subject matter of your second topic is different enough to warrant a second topic. This topic should be enough.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by fth0 »

mpack wrote:
fth0 wrote:AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format.
Are you sure? Or am I misinterpreting your statement? I know for a fact that VirtualBox supports stream compressed VMDK [...]
Yes, VirtualBox supports the streamOptimized VMDK variant (like the OP already stated themselves in the OP ;)), but that's a compression inside the (inner) VMDK format, not inside the (outer) OVF/OVA format. Let me explain (and be pedantic, too ;)):

When exporting an appliance, VirtualBox first converts the virtual disk images to the VMDK variant streamOptimized, which internally uses the DEFLATE compression (RFC 1951). Afterwards, VirtualBox does not compress the VMDK file with gzip (RFC 1952, "GZIP File Format Specification"), which would be the optional compression specified by the OVF/OVA format.
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

Post by fth0 »

A few additional comments on details:
adrelanos wrote:Gzip compression of VMDK files seems to be supported [...] in VirtualBox source code.
No, but DEFLATE compression as part of the VMDK variant streamOptimized is, and .tar.gz packing for the OCI Appliance format.
adrelanos wrote:How to create a GZIP compressed OVA file?
An OVA file is defined to be a tar file, so the only compatible way is to gzip it afterwards.
adrelanos wrote:Why do you think that VirtualBox can import `.ova`s that are actually `.tar.gz` files?
AFAIK, you are the only one thinking that. (On a side note, I thought that too until today, when I learnt that .ova files are indeed uncompressed tar files, thanks to you. ;))
adrelanos wrote:Is this a bug or am I using it wrong?
You're using it wrong: OVF/OVA appliances must have an .ovf or .ova file extension. OCI appliances must have a .tar.gz file extension.
Post Reply