Page 1 of 1

Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Posted: 20. Jun 2017, 08:45
by socratis

The public discussion for the topic is at:
VirtualBox Programming » VirtualBox OSE » [Discussion] Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9


Introduction

The instructions for building VirtualBox on OSX, are quite outdated. There have been significant changes since the 5.x.x series, mainly that if you are building VirtualBox on anything greater than 10.9 (the min OSX supported by VirtualBox), Xcode 6.2 is required, and not Xcode 4.1 as stated in the instructions. Signs of this can be found all over the "src/Config.kmk" if you search for "NEW_XCODE".

There are several version of different how-to floating around. Most of them talk about using pre-built tools from both "MacPorts" and "Homebrew". I did not like the approach of having two package managers, potentially conflicting each other, so I decided to focus on having only "MacPorts" installed in the system. It seems to me that it is the least intrusive, but opinions differ. At some later point, I might do the whole exercise with only "Homebrew". Unless you want to try it and offer your insight...

After a lot of trials and errors, a lot of back and forth, I finally have nailed the instructions for building VirtualBox on OSX >= 10.9. The advice so far was that the devs use a different build system and not the one that was in the official instructions, so I tried to create a set of instructions for a clean system. I used a clean VM for each major OSX version, to make sure there were no dependencies or anything else interfering with the process. And yes, I would revert to the basic install after each trial and error...

The biggest changes that have to be made in the instructions are:
  • No need for Xcode 3.2 or 4.1.
  • No need to extract parts of Xcode.
  • No need in fact for any installed Xcode in 10.9, .10, .11. You can get away just by having the appropriate Xcode DMGs available. Unfortunatelly this does NOT work for 10.12 and 10.13, due to changes in the Xcode format (XIP instead of DMG).
  • No need for the complete Qt installation. It actually complicates/breaks things.
  • A slightly modified "configure". Tried to keep changes at a minimum, hoping that they don't break anything.
  • The changes to include 10.12 as a supported OS have already been merged in revisions 67198 and 67280.
  • The changes to include 10.13 as a supported OS have already been merged in revision 68477.
  • No 32-bit support.
  • No support for anything less than 10.9.
The last two (32-bits and < 10.9) are pretty much the status quo since the 5.0.x days, so that part doesn't break anything, it just needs to be updated in the instructions to reflect RealLife™. Things I haven't resolved (yet):
  • Package the build as a standalone package.
  • Run the test build alongside an existing system w/o installing it, potentially w/o installing the custom KEXTs.
Prerequisites
  1. OSX 10.9 up to 10.13, (preferably up to date), either a physical system, or a VM.
  2. Java 6 runtime (JavaForOSX.dmg).
  3. Xcode_#.#.dmg, OSX specific. Needed for installing and maintaining MacPorts. No need to install it, unless you already have it installed. If you install Xcode in your system, you might need to adjust the paths that point to Xcode.
    • [*]For OSX 10.9  you need Xcode:  5.1.1 - 6.2
      [*]For OSX 10.10 you need Xcode:  6.2   - 7.2.1
      [*]For OSX 10.11 you need Xcode:  7.0.1 - 8.2.1
      [*]For OSX 10.12 you need Xcode:  8.0   - 8.3.3
      [*]For OSX 10.13 you need Xcode:  8.2.1 - 9.4.1
  4. Xcode_6.2.dmg. Needed for compiling VirtualBox. No need to install it, unless you already have it installed. If you install Xcode 6.2 in your system, you might need to adjust the paths that point to that.
  5. MacPorts-a.b.c-10.##-OSXName.pkg, OSX specific. MacPorts is at version 2.5.3 as of 2018-07-05, available for 10.9.x-10.13.x. So, for 10.11.6 for example, you'll have to get MacPorts-2.5.3-10.11-ElCapitan.pkg.
  6. VirtualBox source code.
Step-by-step instructions
  1. Get Java for OSX
    Install Java support from https://support.apple.com/kb/DL1572. Needed at the very end of the building process, but you might as well get it over with.
    hdiutil attach JavaForOSX.dmg
    time sudo installer -pkg /Volumes/Java\ for\ OS\ X\ 2015-001/JavaForOSX.pkg -target /
    hdiutil detach /Volumes/Java\ for\ OS\ X\ 2015-001/
    or
    hdiutil attach JavaForOSX.dmg
    time sudo installer -pkg /Volumes/Java\ for\ OS\ X\ 2017-001/JavaForOSX.pkg -target /
    hdiutil detach /Volumes/Java\ for\ OS\ X\ 2017-001/
  2. Attach/install Xcode for MacPorts
    Xcode is a peculiar beast. You must have Xcode 6.2 available in order to build VirtualBox. Otherwise you get unlinked symbol(s) errors on 10.10-10.12 if you have anything higher than Xcode 6.2. But, more on that later.

    On the other hand, you must have an Xcode-<OSX_specific> in order to install and update MacPorts. Otherwise MacPorts chokes (complains and aborts). After you've installed MacPorts, you don't need the Xcode-<OSX_specific> anymore.
    hdiutil attach Xcode_#.#.dmg
    sudo xcode-select -s /Volumes/Xcode/Xcode.app
    sudo xcodebuild -license accept
    
    Note for 10.12, 10.13
    The Xcode required is Xcode > 8, which comes in a non-mountable .XIP format. You have to extract it and install it. No workaround available at this time. In that case, after you've installed Xcode (in /Applications), do:
    sudo xcode-select -s /Applications/Xcode.app
    sudo xcodebuild -license accept
    
  3. Install MacPorts and the required packages
    Download the MacPorts-a.b.c-10.##-OSXName.pkg, OSX-specific package.
    time sudo installer -pkg MacPorts-a.b.c-10.##-OSXName.pkg -target /
    echo export PATH=\"/opt/local/bin:/opt/local/sbin:\$PATH\" > ~/.profile
    source ~/.profile
    
    time sudo port selfupdate
    time sudo port -N install libidl +universal doxygen texlive texlive-latex-extra \
                              texlive-fonts-extra cdrtools openssl qt56 subversion \
                              docbook-xml docbook-xsl-nons
    This is a long process, mainly because it has to download GB of data. On my system it took 2 hrs and more that 7.5 GB.

    You can safely ignore the messages:
    Warning: System headers do not appear to be installed. Most ports should build correctly, but if you experience problems due to a port depending on system headers, please file a ticket at https://trac.macports.org.
    Warning: You can install them as part of the Xcode Command Line Tools package by running `xcode-select --install'.
     Edit: Added bonus with the 'qt56' package you get the QtLinquist, for you translators out there. 
  4. Get the VirtualBox source code
    For a new installation:
    Or, if you're upgrading:
    svn upgrade /Users/Shared/vbox
    svn update /Users/Shared/vbox
  5. Changes to the source code
    Save the following file as "~/Patch_VirtualBox.diff":
    • Code: Select all

      Index: configure
      ===================================================================
      --- configure	(revision 67459)
      +++ configure	(working copy)
      @@ -153,7 +153,7 @@
       LIBVPX="-lvpx"
       PKGCONFIG="`which_wrapper pkg-config`"
       PYTHONDIR="/usr /usr/local"
      -QT5DIR="/usr/lib/qt5 /usr/share/qt5 /usr/lib64/qt5 /usr /usr/local"
      +QT5DIR="/usr/lib/qt5 /usr/share/qt5 /usr/lib64/qt5 /usr /usr/local /opt/local/libexec/qt5"
       QT5DIR_PKGCONFIG=1
       QT5MAJ=5
       QT5MIN=6
      @@ -1515,7 +1515,7 @@
           # Now try the user provided directory and some of the standard directories.
           QT_TRIES="$QT5DIR /System/Library /Library"
           for t in $QT_TRIES; do
      -      if [ -f "$t/Frameworks/QtCore.framework/QtCore" ]; then
      +      if [ -f "$t/lib/QtCore.framework/QtCore" ]; then
               PATH_SDK_QT5="$t"
               break
             fi
      @@ -1523,8 +1523,8 @@
           # Add the necessary params for building the test application
           if [ -n "$PATH_SDK_QT5" ]; then
             foundqt5=1
      -      INCQT5=-I$PATH_SDK_QT5/Frameworks/QtCore.framework/Headers
      -      LIBQT5=-F$PATH_SDK_QT5/Frameworks
      +      INCQT5=-I$PATH_SDK_QT5/lib/QtCore.framework/Headers
      +      LIBQT5=-F$PATH_SDK_QT5/lib
             FLGQT5="-framework QtCore"
           else
             log_failure "Qt5 framework not found (can be disabled using --disable-qt)"
      @@ -1605,9 +1605,9 @@
               if [ "$OS" = "darwin" ]; then
                 # Successful build & run the test application so add the necessary
                 # params to AutoConfig.kmk:
      -          cnf_append "PATH_SDK_QT5_INC" "$PATH_SDK_QT5/Frameworks"
      -          cnf_append "PATH_SDK_QT5_LIB" "$PATH_SDK_QT5/Frameworks"
      -          cnf_append "PATH_SDK_QT5" "$PATH_SDK_QT5/Frameworks"
      +          cnf_append "PATH_SDK_QT5_INC" "$PATH_SDK_QT5/inc"
      +          cnf_append "PATH_SDK_QT5_LIB" "$PATH_SDK_QT5/lib"
      +          cnf_append "PATH_SDK_QT5" "$PATH_SDK_QT5"
                 # Check for the moc tool in the Qt directory found & some standard
                 # directories.
                 for q in $PATH_SDK_QT5 /usr /Developer/Tools/Qt; do
      
    Then apply that patch to the source code tree:
    svn patch ~/Patch_VirtualBox.diff /Users/Shared/vbox/[/color]
    If you want to build the "UserManual.pdf", you need to have a file called "LocalConfig.kmk" in the top of your source directory. At a minimum, this file should contain the following:
    • Code: Select all

      # The documentation
      VBOX_WITH_DOCS = 1
      
      # Include documentation in the package
      VBOX_WITH_DOCS_PACKING = 1
      
      # Starting with build https://www.virtualbox.org/changeset/72949/vbox/trunk/doc/manual/Config.kmk
      # you'll need to modify the DocBook paths.
      #
      # VBOX_PATH_DOCBOOK        = /usr/share/xml/docbook/stylesheet/docbook-xsl
        VBOX_PATH_DOCBOOK        = /opt/local/share/xsl/docbook-xsl-nons
      # VBOX_PATH_DOCBOOK_DTD    = /usr/share/xml/docbook/schema/dtd/4.5
        VBOX_PATH_DOCBOOK_DTD    = /opt/local/share/xml/docbook/4.5
      
  6. Attach Xcode 6.2 for building VirtualBox
    Unmount "Xcode_#.#" and mount "Xcode_6.2.dmg".
    hdiutil detach /Volumes/Xcode/
    hdiutil attach Xcode_6.2.dmg
    sudo xcode-select -s /Volumes/Xcode/Xcode.app
    sudo xcodebuild -license accept
    
  7. Build VirtualBox
    If you don't want to be doing a clean install every time, you just need the last command ("time kmk").
    cd /Users/Shared/vbox
    sudo rm -r /Users/Shared/vbox/out/
    ./configure --disable-hardening --with-xcode-dir=/Volumes/Xcode/Xcode.app
    source ./env.sh
    
    time kmk
    
  8. Disable SIP
    You won't be able to load the KEXTs required for VirtualBox to run, unless you take care of the System Integrity Protection (SIP) which prevents unsigned KEXTs from being loaded.
    • OSX 10.9
      It just works. No SIP in 10.9.
    • OSX 10.10
      • For a VM, set:
        VBoxManage setextradata "<vm>" VBoxInternal2/EfiBootArgs "kext-dev-mode=1 usb=0x800 keepsyms=1 -v -serial=0x1"
      • For a real system, set:
        sudo nvram boot-args="kext-dev-mode=1"
    • OSX ≥ 10.11
      • For a VM (then go to "For both"):
        F12                          (multiple times at boot to enter the EFI shell)
        Boot Manager
        EFI Internal shell
        [quote]FS2:                         (to switch to the EFI partition)
        ls                           (to verify that "com.apple.recovery.boot" is there)
        cd com.apple.recovery.boot
        boot.efi                     (will boot you to the installation phase)[/quote]
      • For a real system (then go to "For both"):
        Cmd-R                        (to enter recovery mode)
      • For both:
        Select Utilities » Terminal:
        csrutil disable
        reboot
    After you've disabled SIP, you can load the VirtualBox KEXTs:
    /Users/Shared/vbox/out/darwin.amd64/release/dist/loadall.sh
  9. Run VirtualBox
    If everything is working, your VirtualBox can be launched by:
    /Users/Shared/vbox/out/darwin.amd64/release/dist/VirtualBox.app/Contents/MacOS/VirtualBox

Re: Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Posted: 21. Aug 2018, 08:02
by socratis
Xcode downloads

For your convenience and reference, here are the different Xcode versions mentioned in this thread. You should always try to get the latest supported version for your up-to-date OSX version. All links point to the Apple Developers Website, you will need to have an active Apple Developer ID: MacPorts downloads

For your convenience and reference, here are the different Xcode versions mentioned in this thread. You should always try to get the latest supported version for your up-to-date OSX version. Last update: 2018-08-21