Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Post by **socratis** » 20. Jun 2017, 08:45

The public discussion for the topic is at:
VirtualBox Programming » VirtualBox OSE » [Discussion] Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Introduction

The instructions for building VirtualBox on OSX, are quite outdated. There have been significant changes since the 5.x.x series, mainly that if you are building VirtualBox on anything greater than 10.9 (the min OSX supported by VirtualBox), Xcode 6.2 is required, and not Xcode 4.1 as stated in the instructions. Signs of this can be found all over the "src/Config.kmk" if you search for "NEW_XCODE".

There are several version of different how-to floating around. Most of them talk about using pre-built tools from both "MacPorts" and "Homebrew". I did not like the approach of having two package managers, potentially conflicting each other, so I decided to focus on having only "MacPorts" installed in the system. It seems to me that it is the least intrusive, but opinions differ. At some later point, I might do the whole exercise with only "Homebrew". Unless you want to try it and offer your insight...

After a lot of trials and errors, a lot of back and forth, I finally have nailed the instructions for building VirtualBox on OSX >= 10.9. The advice so far was that the devs use a different build system and not the one that was in the official instructions, so I tried to create a set of instructions for a clean system. I used a clean VM for each major OSX version, to make sure there were no dependencies or anything else interfering with the process. And yes, I would revert to the basic install after each trial and error...

The biggest changes that have to be made in the instructions are:

No need for Xcode 3.2 or 4.1.
No need to extract parts of Xcode.
No need in fact for any installed Xcode in 10.9, .10, .11. You can get away just by having the appropriate Xcode DMGs available. Unfortunatelly this does NOT work for 10.12 and 10.13, due to changes in the Xcode format (XIP instead of DMG).
No need for the complete Qt installation. It actually complicates/breaks things.
A slightly modified "configure". Tried to keep changes at a minimum, hoping that they don't break anything.
The changes to include 10.12 as a supported OS have already been merged in revisions 67198 and 67280.
The changes to include 10.13 as a supported OS have already been merged in revision 68477.
No 32-bit support.
No support for anything less than 10.9.

The last two (32-bits and < 10.9) are pretty much the status quo since the 5.0.x days, so that part doesn't break anything, it just needs to be updated in the instructions to reflect RealLife™. Things I haven't resolved (yet):

Package the build as a standalone package.
Run the test build alongside an existing system w/o installing it, potentially w/o installing the custom KEXTs.

Prerequisites

OSX 10.9 up to 10.13, (preferably up to date), either a physical system, or a VM.
Java 6 runtime (JavaForOSX.dmg).

Xcode_#.#.dmg, OSX specific. Needed for installing and maintaining MacPorts. No need to install it, unless you already have it installed. If you install Xcode in your system, you might need to adjust the paths that point to Xcode.

[*]For OSX 10.9  you need Xcode:  5.1.1 - 6.2
[*]For OSX 10.10 you need Xcode:  6.2   - 7.2.1
[*]For OSX 10.11 you need Xcode:  7.0.1 - 8.2.1
[*]For OSX 10.12 you need Xcode:  8.0   - 8.3.3
[*]For OSX 10.13 you need Xcode:  8.2.1 - 9.4.1

Xcode_6.2.dmg. Needed for compiling VirtualBox. No need to install it, unless you already have it installed. If you install Xcode 6.2 in your system, you might need to adjust the paths that point to that.
MacPorts-a.b.c-10.##-OSXName.pkg, OSX specific. MacPorts is at version 2.5.3 as of 2018-07-05, available for 10.9.x-10.13.x. So, for 10.11.6 for example, you'll have to get MacPorts-2.5.3-10.11-ElCapitan.pkg.
VirtualBox source code.

Step-by-step instructions

Get Java for OSX
Install Java support from https://support.apple.com/kb/DL1572. Needed at the very end of the building process, but you might as well get it over with.

hdiutil attach JavaForOSX.dmg
time sudo installer -pkg /Volumes/Java\ for\ OS\ X\ 2015-001/JavaForOSX.pkg -target /
hdiutil detach /Volumes/Java\ for\ OS\ X\ 2015-001/

or

hdiutil attach JavaForOSX.dmg
time sudo installer -pkg /Volumes/Java\ for\ OS\ X\ 2017-001/JavaForOSX.pkg -target /
hdiutil detach /Volumes/Java\ for\ OS\ X\ 2017-001/

Attach/install Xcode for MacPorts
Xcode is a peculiar beast. You must have Xcode 6.2 available in order to build VirtualBox. Otherwise you get unlinked symbol(s) errors on 10.10-10.12 if you have anything higher than Xcode 6.2. But, more on that later.

On the other hand, you must have an Xcode-<OSX_specific> in order to install and update MacPorts. Otherwise MacPorts chokes (complains and aborts). After you've installed MacPorts, you don't need the Xcode-<OSX_specific> anymore.
```
hdiutil attach Xcode_#.#.dmg
sudo xcode-select -s /Volumes/Xcode/Xcode.app
sudo xcodebuild -license accept
```
Note for 10.12, 10.13
The Xcode required is Xcode > 8, which comes in a non-mountable .XIP format. You have to extract it and install it. No workaround available at this time. In that case, after you've installed Xcode (in /Applications), do:
```
sudo xcode-select -s /Applications/Xcode.app
sudo xcodebuild -license accept
```

Install MacPorts and the required packages
Download the MacPorts-a.b.c-10.##-OSXName.pkg, OSX-specific package.

time sudo installer -pkg MacPorts-a.b.c-10.##-OSXName.pkg -target /
echo export PATH=\"/opt/local/bin:/opt/local/sbin:\$PATH\" > ~/.profile
source ~/.profile

time sudo port selfupdate
time sudo port -N install libidl +universal doxygen texlive texlive-latex-extra \
                          texlive-fonts-extra cdrtools openssl qt56 subversion \
                          docbook-xml docbook-xsl-nons

This is a long process, mainly because it has to download GB of data. On my system it took 2 hrs and more that 7.5 GB.

You can safely ignore the messages:

Warning: System headers do not appear to be installed. Most ports should build correctly, but if you experience problems due to a port depending on system headers, please file a ticket at https://trac.macports.org.
Warning: You can install them as part of the Xcode Command Line Tools package by running `xcode-select --install'.

Edit: Added bonus with the 'qt56' package you get the QtLinquist, for you translators out there.

Get the VirtualBox source code
For a new installation:

svn co http://www.virtualbox.org/svn/vbox/trunk /Users/Shared/vbox

Or, if you're upgrading:

svn upgrade /Users/Shared/vbox
svn update /Users/Shared/vbox

Changes to the source code
Save the following file as "~/Patch_VirtualBox.diff":

Code: Select all

Index: configure
===================================================================
--- configure	(revision 67459)
+++ configure	(working copy)
@@ -153,7 +153,7 @@
 LIBVPX="-lvpx"
 PKGCONFIG="`which_wrapper pkg-config`"
 PYTHONDIR="/usr /usr/local"
-QT5DIR="/usr/lib/qt5 /usr/share/qt5 /usr/lib64/qt5 /usr /usr/local"
+QT5DIR="/usr/lib/qt5 /usr/share/qt5 /usr/lib64/qt5 /usr /usr/local /opt/local/libexec/qt5"
 QT5DIR_PKGCONFIG=1
 QT5MAJ=5
 QT5MIN=6
@@ -1515,7 +1515,7 @@
     # Now try the user provided directory and some of the standard directories.
     QT_TRIES="$QT5DIR /System/Library /Library"
     for t in $QT_TRIES; do
-      if [ -f "$t/Frameworks/QtCore.framework/QtCore" ]; then
+      if [ -f "$t/lib/QtCore.framework/QtCore" ]; then
         PATH_SDK_QT5="$t"
         break
       fi
@@ -1523,8 +1523,8 @@
     # Add the necessary params for building the test application
     if [ -n "$PATH_SDK_QT5" ]; then
       foundqt5=1
-      INCQT5=-I$PATH_SDK_QT5/Frameworks/QtCore.framework/Headers
-      LIBQT5=-F$PATH_SDK_QT5/Frameworks
+      INCQT5=-I$PATH_SDK_QT5/lib/QtCore.framework/Headers
+      LIBQT5=-F$PATH_SDK_QT5/lib
       FLGQT5="-framework QtCore"
     else
       log_failure "Qt5 framework not found (can be disabled using --disable-qt)"
@@ -1605,9 +1605,9 @@
         if [ "$OS" = "darwin" ]; then
           # Successful build & run the test application so add the necessary
           # params to AutoConfig.kmk:
-          cnf_append "PATH_SDK_QT5_INC" "$PATH_SDK_QT5/Frameworks"
-          cnf_append "PATH_SDK_QT5_LIB" "$PATH_SDK_QT5/Frameworks"
-          cnf_append "PATH_SDK_QT5" "$PATH_SDK_QT5/Frameworks"
+          cnf_append "PATH_SDK_QT5_INC" "$PATH_SDK_QT5/inc"
+          cnf_append "PATH_SDK_QT5_LIB" "$PATH_SDK_QT5/lib"
+          cnf_append "PATH_SDK_QT5" "$PATH_SDK_QT5"
           # Check for the moc tool in the Qt directory found & some standard
           # directories.
           for q in $PATH_SDK_QT5 /usr /Developer/Tools/Qt; do

Then apply that patch to the source code tree:

svn patch ~/Patch_VirtualBox.diff /Users/Shared/vbox/[/color]

If you want to build the "UserManual.pdf", you need to have a file called "LocalConfig.kmk" in the top of your source directory. At a minimum, this file should contain the following:

Code: Select all

# The documentation
VBOX_WITH_DOCS = 1

# Include documentation in the package
VBOX_WITH_DOCS_PACKING = 1

# Starting with build https://www.virtualbox.org/changeset/72949/vbox/trunk/doc/manual/Config.kmk
# you'll need to modify the DocBook paths.
#
# VBOX_PATH_DOCBOOK        = /usr/share/xml/docbook/stylesheet/docbook-xsl
  VBOX_PATH_DOCBOOK        = /opt/local/share/xsl/docbook-xsl-nons
# VBOX_PATH_DOCBOOK_DTD    = /usr/share/xml/docbook/schema/dtd/4.5
  VBOX_PATH_DOCBOOK_DTD    = /opt/local/share/xml/docbook/4.5

Attach Xcode 6.2 for building VirtualBox
Unmount "Xcode_#.#" and mount "Xcode_6.2.dmg".

hdiutil detach /Volumes/Xcode/
hdiutil attach Xcode_6.2.dmg
sudo xcode-select -s /Volumes/Xcode/Xcode.app
sudo xcodebuild -license accept

Build VirtualBox
If you don't want to be doing a clean install every time, you just need the last command ("time kmk").

cd /Users/Shared/vbox
sudo rm -r /Users/Shared/vbox/out/
./configure --disable-hardening --with-xcode-dir=/Volumes/Xcode/Xcode.app
source ./env.sh

time kmk

Disable SIP
You won't be able to load the KEXTs required for VirtualBox to run, unless you take care of the System Integrity Protection (SIP) which prevents unsigned KEXTs from being loaded.

OSX 10.9
It just works. No SIP in 10.9.

OSX 10.10

For a VM, set:

VBoxManage setextradata "<vm>" VBoxInternal2/EfiBootArgs "kext-dev-mode=1 usb=0x800 keepsyms=1 -v -serial=0x1"

For a real system, set:
```
sudo nvram boot-args="kext-dev-mode=1"
```

OSX ≥ 10.11

For a VM (then go to "For both"):

F12                          (multiple times at boot to enter the EFI shell)
Boot Manager
EFI Internal shell
[quote]FS2:                         (to switch to the EFI partition)
ls                           (to verify that "com.apple.recovery.boot" is there)
cd com.apple.recovery.boot
boot.efi                     (will boot you to the installation phase)[/quote]

For a real system (then go to "For both"):

Cmd-R                        (to enter recovery mode)

For both:
Select Utilities » Terminal:
```
csrutil disable
reboot
```

After you've disabled SIP, you can load the VirtualBox KEXTs:

/Users/Shared/vbox/out/darwin.amd64/release/dist/loadall.sh

Run VirtualBox
If everything is working, your VirtualBox can be launched by:

/Users/Shared/vbox/out/darwin.amd64/release/dist/VirtualBox.app/Contents/MacOS/VirtualBox

Post by **socratis** » 21. Aug 2018, 08:02

Xcode downloads

For your convenience and reference, here are the different Xcode versions mentioned in this thread. You should always try to get the latest supported version for your up-to-date OSX version. All links point to the Apple Developers Website, you will need to have an active Apple Developer ID:

MacPorts downloads

For your convenience and reference, here are the different Xcode versions mentioned in this thread. You should always try to get the latest supported version for your up-to-date OSX version.

Last update: 2018-08-21

Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9

Re: Build VirtualBox ≥ 5.1.2 on OSX ≥ 10.9