How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

[adrelanos]

How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

What's the purpose? VMDKs in exported VirtualBox OVAs are compressed, yes, but better compression is possible by using tar with maximum gzip compression. However, a .tar.gz file that contains an .ova is harder to use and takes longer to import (requires untar + import). Therefore I would like to have an .ova file that contains a gzip compressed disk image .vmdk. This would have the advantage that double clicking an .ova file is still associated with VirtualBox. But if it's a .tar.gz that contains an .ova, that won't work.

If this is impossible using VirtualBox GUI as well as vboxmanage command line, how could one accomplish the .ova using gzip compression for its .vmdk images?

What I am not looking for is vboxmanage modifymedium --compact / zerofree because that is already discussed elsewhere, it's great, and I am already using that. Additional gzip compression on top of it would be even better.

Gzip compression of VMDK files seems to be supported in both the OVA standard as well as in VirtualBox source code.

Quote OVA standard:

Each file referenced by a File element may be compressed using gzip (see RFC1952). When a File
element is compressed using gzip, the ovf:compression attribute shall be set to “gzip”.

VMware has a bug OVF Package with compressed disks is currently not supported for OVF import (53260). It may seem besides the point but this further indicates that this is possible in theory.

VirtualBox source file tarvfs.cpp mentions gzip several times. So does ApplianceImplImport.cpp and ApplianceImplExport.cpp. Also VirtualBox IAppliance Interface Reference mentions gzip.

VirtualBox source code folder contains a file src/VBox/ValidationKit/tests/api/tdAppliance1-t5.ova which is I guess used as a unit test and that file describes gzip compression.

Code: Select all   Expand viewCollapse view

    <File ovf:compression="gzip" ovf:href="tdAppliance1-t5-disk1.vmdk.gz" ovf:id="file1" ovf:size="509"/>


That further indicates that VirtualBox supports this. The only thing that I couldn't find out after hours of internet and source code research is how to create an OVA that uses gzip compression for the disk images it includes. One could:

1. export to .ovf instead,
2. manually or with a script compress the .vmdk with gzip,
3. then edit the .ovf
4. create a tar archive, the actual .ova

That might work? But seems rather cumbersome. Is there an existing tool which can help here?

[scottgus1]

One problem you'll have is that if you deviate from the OVF standard in making the OVA, then you won't be making an OVA anymore. You'll be making an 'ADRELANOS', that is, a private distribution method not compatible with OVF.

These options are available with the "vboxmanage export" command:

export <machines> --output|-o <name>.<ovf/ova/tar.gz>
[--legacy09|--ovf09|--ovf10|--ovf20|--opc10]
[--manifest]
[--options manifest|iso|nomacs|nomacsbutnat]
[--vsys <number of virtual system>]
[--vmname <name>]
[--product <product name>]
[--producturl <product url>]
[--vendor <vendor name>]
[--vendorurl <vendor url>]
[--version <version info>]
[--description <description info>]
[--eula <license text>]
[--eulafile <filename>]

Sections 1.14.3 and 8.11 cover the meaning of some of the options. There is a tar.gz option, so maybe what you want is already there. Or this may tar the whole OVA file, I don't know.

[fth0]

AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format. The easiest way to check that is to search for "ovf:compression", which is neither written nor read nor interpreted anywhere in the VirtualBox 6.1.38 sources. You can also search for "ovf:size" to find the places where the compression option would have to be dealt with. In consequence, you'd have to implement the use of gzip compression yourself if you really wanted it.

What you've found are usages of the .tar.gz format, for example for the OCI Appliance import/export.

[mpack]

AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format.

Are you sure? Or am I misinterpreting your statement? I know for a fact that VirtualBox supports stream compressed VMDK on import. In fact it's not restricted to import, it's a part of the normal VMDK handling: you can build a VM around a stream compressed VMDK and it will work, but VirtualBox will use a difference image to handle writes.

What I don't know is whether VirtualBox uses compression on the OVA files it creates. It includes the zlib library so I'd be surprised if not. [EDIT] I checked the vmdk resulting from an exported appliance (OVA). The VMDK was stream compressed according to CloneVDI.

pedantic note: gzip is a piece of software, not a stream format. AFAIK the stream format is covered by RFC 1951 and is used by several tools, e.g. PNG. I imagine it also includes gzip.

[adrelanos]

My main question:
How to create a GZIP compressed OVA file?


Not "create OVA first, then wrap it into a TAR.GZ." please. A "directly" gzip compressed OVA.

An OVA file is supposedly just a TAR file and in theory should support GZIP compression. (See below for references.)

Create OVA from Command Line using TAR - GZIP - Broken

Trying to import a VirtualBox OVA that I created manually using the command line, I am getting a VERR_TAR_BAD_CHKSUM_FIELD error.

This was my command to manually create the OVA from command line using tar (broken):

Code: Select all   Expand viewCollapse view

tar --gzip --create --verbose --directory=/home/user/temp --file temp.ova temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf


temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf

Trying to import... No... Failing...

Code: Select all   Expand viewCollapse view

vboxmanage import temp.ova ; echo $?


Progress state: VBOX_E_IPRT_ERROR
VBoxManage: error: Appliance read failed
VBoxManage: error: Error reading OVA '/home/user/temp/temp.ova' (VERR_TAR_BAD_CHKSUM_FIELD)
VBoxManage: error: Details: code VBOX_E_IPRT_ERROR (0x80bb0005), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg*)" at line 471 of file VBoxManageAppliance.cpp

Create OVA from Command Line using TAR - Functional - But lacking GZIP Compression

If I drop --gzip from tar parameters it's functional.

This was my command to manually create the OVA from command line using tar (functional):

Code: Select all   Expand viewCollapse view

tar --create --verbose --directory=/home/user/temp --file temp.ova temp-disk001.vmdk temp-disk002.vmdk temp.mf temp.ovf


Then import works.

Code: Select all   Expand viewCollapse view

vboxmanage import temp.ova ; echo $?


[...]
0

Notes:

For the error VERR_TAR_BAD_CHKSUM_FIELD I have found the related VirtualBox source code. [1] [2]

What tar checksum does VirtualBox expect? I tried to learn more about if tar checksums exist and how they work but I couldn't figure out anything yet. See also Does the tar format support checksums?

Also tar parameter --format=ustar did not help.

• What I am not looking for is vboxmanage modifymedium --compact / zerofree because that is already discussed elsewhere, it's great, and I am already using that. Additional gzip compression on top of it would be even better.
• What my main goal? A gzip compressed `.ova` VirtualBox appliances that can be imported directly into VirtualBox without an additional `.tar.gz`.
• Aren't VirtualBox ova's compressed anyhow? No. These are just tar files. Not gzip or xz compressed archives.
• But the `.vmdk` files in `.ova` files are compressed? Yes, but the compression is limited. By using an additional `.tar.gz` with maximum compression, the `.ova` gets smaller.
• Why not use an additional `.tar.gz` file? Compression, speed, usability, simplicity.
• Why do you think that VirtualBox can import `.ova`s that are actually `.tar.gz` files? In How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export? I posted various references to gzip being mentioned in VirtualBox source code and OVA specification.
• Is How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export? a duplicate? No. That one is about gzip compression of disk images inside OFV / OVA. This forum thread is about gzip compressed OVA. It's not exactly the same. It's a different approach that might accomplish the same end goal of better OVA compression.
• Why am I mentioning all of this? Because most questions that I've seen go into direction of `vboxmanage modifymedium --compact` / `zerofree`. Therefore I wanted to be very specific here: An `.ova` file that is gzip compressed.

[1]

Code: Select all   Expand viewCollapse view

/**
 * Validates the TAR header.
 *
 * @returns VINF_SUCCESS if valid, VERR_TAR_ZERO_HEADER if all zeros, and
 *          the appropriate VERR_TAR_XXX otherwise.
 * @param   pTar                The TAR header.
 * @param   penmType            Where to return the type of header on success.
 */
static int rtZipTarHdrValidate(PCRTZIPTARHDR pTar, PRTZIPTARTYPE penmType)
{
    /*
     * Calc the checksum first since this enables us to detect zero headers.
     */
    int32_t i32ChkSum;
    int32_t i32ChkSumSignedAlt;
    if (rtZipTarCalcChkSum(pTar, &i32ChkSum, &i32ChkSumSignedAlt))
        return VERR_TAR_ZERO_HEADER;

    /*
     * Read the checksum field and match the checksums.
     */
    int64_t i64HdrChkSum;
    int rc = rtZipTarHdrFieldToNum(pTar->Common.chksum, sizeof(pTar->Common.chksum), true /*fOctalOnly*/, &i64HdrChkSum);
    if (RT_FAILURE(rc))
        return VERR_TAR_BAD_CHKSUM_FIELD;
    if (   i32ChkSum          != i64HdrChkSum
        && i32ChkSumSignedAlt != i64HdrChkSum) /** @todo test this */
        return VERR_TAR_CHKSUM_MISMATCH;


[2]

Code: Select all   Expand viewCollapse view

/*
 * Copyright (C) 2010-2020 Oracle Corporation
 *
 * This file is part of VirtualBox Open Source Edition (OSE), as
 * available from http://www.virtualbox.org. This file is free software;
 * you can redistribute it and/or modify it under the terms of the GNU
 * General Public License (GPL) as published by the Free Software
 * Foundation, in version 2 as it comes in the "COPYING" file of the
 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
 *
 * The contents of this file may alternatively be used under the terms
 * of the Common Development and Distribution License Version 1.0
 * (CDDL) only, as it comes in the "COPYING.CDDL" file of the
 * VirtualBox OSE distribution, in which case the provisions of the
 * CDDL are applicable instead of those of the GPL.
 *
 * You may elect to license modified versions of this file under the
 * terms and conditions of either the GPL or the CDDL or both.
 */


Create OVA from Command Line using VBoxManage - GZIP - Broken

This is might actually be a different approach from above. The following might create (if that was working) an ova that is wrapped into a separate tar.gz?

Syntax shown when running "VBoxManage export" (built-in help).

Code: Select all   Expand viewCollapse view

VBoxManage export           <machines> --output|-o <name>.<ovf/ova/tar.gz>

functional:

Code: Select all   Expand viewCollapse view

vboxmanage export test --output test.ova

So a uncompressed ova can be created using vboxmanage.

broken:

Code: Select all   Expand viewCollapse view

vboxmanage export test --output test.tar.gz

Code: Select all   Expand viewCollapse view

vboxmanage export test --output test.ova.tar.gz

In both cases...

VBoxManage: error: Appliance file must have .ovf or .ova extension
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance, callee nsISupports
VBoxManage: error: Context: "Write(Bstr(strOvfFormat).raw(), ComSafeArrayAsInParam(options), Bstr(pszAbsFilePath).raw(), progress.asOutParam())" at line 1934 of file VBoxManageAppliance.cpp

Is this a bug or am I using it wrong?

[mpack]

One problem you'll have is that if you deviate from the OVF standard in making the OVA, then you won't be making an OVA anymore.

I think this is compatible with the OVF standard. Here's what I found out:

1. I went to https://en.wikipedia.org/wiki/Open_Virt ... ion_Format to learn more about OVF
2. it links to the specification https://www.dmtf.org/standards/ovf
3. DSP0243 2.1.1 Open Virtualization Format Specification 27 Aug 2015 Standard View
Interesting. Click.
4. https://www.dmtf.org/sites/default/file ... _2.1.1.pdf
5. Title of pdf is "Open Virtualization Format Specification", good.
6. Search of "gzip" inside the pdf.
7. Found this:

Each file referenced by a File element may be compressed using gzip (see RFC1952). When a File
536 element is compressed using gzip, the ovf:compression attribute shall be set to “gzip”.

So in conclusion, OVF standard supports it.

VirtualBox seems to support to import something along gzip too. See references above. Just creation seems to be undocumented.

These options are available with the "vboxmanage export" command:

export <machines> --output|-o <name>.<ovf/ova/tar.gz>
[--legacy09|--ovf09|--ovf10|--ovf20|--opc10]
[--manifest]
[--options manifest|iso|nomacs|nomacsbutnat]
[--vsys <number of virtual system>]
[--vmname <name>]
[--product <product name>]
[--producturl <product url>]
[--vendor <vendor name>]
[--vendorurl <vendor url>]
[--version <version info>]
[--description <description info>]
[--eula <license text>]
[--eulafile <filename>]

Sections 1.14.3 and 8.11 cover the meaning of some of the options. There is a tar.gz option, so maybe what you want is already there. Or this may tar the whole OVA file, I don't know.

That's actually a different approach which I've created a different forum thread for. Hope that's OK. I think it's sufficiently different to warrant a different forum thread. The end goal of better compression while still having OVA files that are easily importable into VirtualBox (double click on OVA) is the same. But the technical approach is a different one. This approach in this forum thread uses gzip on the VMDK which is then to be defined in the OVF ("inner layer"). The other approach in a separate forum thread uses gzip for the OVA ("outer layer").

- This forum thread: How to compress disk images (VMDK) inside an OVA with GZIP during VirtualBox export?

- Alternative approach: {dead link removed}

[mpack]

Cross posting is against the forum rules, and I don't believe the subject matter of your second topic is different enough to warrant a second topic. This topic should be enough.

[fth0]

AFAICS, VirtualBox 6.1 does not support gzip compression for the disk images inside the OVF/OVA format.

Are you sure? Or am I misinterpreting your statement? I know for a fact that VirtualBox supports stream compressed VMDK [...]

Yes, VirtualBox supports the streamOptimized VMDK variant (like the OP already stated themselves in the OP ), but that's a compression inside the (inner) VMDK format, not inside the (outer) OVF/OVA format. Let me explain (and be pedantic, too ):

When exporting an appliance, VirtualBox first converts the virtual disk images to the VMDK variant streamOptimized, which internally uses the DEFLATE compression (RFC 1951). Afterwards, VirtualBox does not compress the VMDK file with gzip (RFC 1952, "GZIP File Format Specification"), which would be the optional compression specified by the OVF/OVA format.

[fth0]

A few additional comments on details:

Gzip compression of VMDK files seems to be supported [...] in VirtualBox source code.

No, but DEFLATE compression as part of the VMDK variant streamOptimized is, and .tar.gz packing for the OCI Appliance format.

How to create a GZIP compressed OVA file?

An OVA file is defined to be a tar file, so the only compatible way is to gzip it afterwards.

Why do you think that VirtualBox can import `.ova`s that are actually `.tar.gz` files?

AFAIK, you are the only one thinking that. (On a side note, I thought that too until today, when I learnt that .ova files are indeed uncompressed tar files, thanks to you. )

Is this a bug or am I using it wrong?

You're using it wrong: OVF/OVA appliances must have an .ovf or .ova file extension. OCI appliances must have a .tar.gz file extension.