Building XNU for macOS High Sierra 10.13

The macOS High Sierra kernel (XNU) source has been released here: source, tarball.

Building XNU requires some patience, and some open source dependencies which are not pre-installed. This post walks through all the steps necessary to run your own custom-built XNU on supported Apple hardware. A VM is recommended, but the following steps will build a kernel suitable for both a VM and bare metal.

TL;DR

There is a Makefile which automates the downloading and building of all prerequisites. You can find the Makefile here, and invoke it like:

$ make -f Makefile.xnudeps xnudeps

Note that this makefile also automates the building of libsyscall which is necessary if you plan to add system calls to XNU. To build libsyscall, you can do this:

$ make -f Makefile.xnudeps libsyscall

$ make -f Makefile.xnudeps libsyscall_install

Manual XNU Building

All of the source for both XNU and required dependencies is available from opensource.apple.com. Here are the manual steps necessary to build XNU:

1. Download the source
• export TARBALLS=https://opensource.apple.com/tarballs
• curl -O ${TARBALLS}/dtrace/dtrace-262.tar.gz
• curl -O ${TARBALLS}/AvailabilityVersions/AvailabilityVersions-32.tar.gz
• curl -O ${TARBALLS}/libplatform/libplatform-161.tar.gz
• curl -O ${TARBALLS}/libdispatch/libdispatch-913.1.6.tar.gz
• curl -O ${TARBALLS}/xnu/xnu-4570.1.46.tar.gz
2. Build CTF tools from dtrace
• tar zxf dtrace-262.tar.gz
• cd dtrace-262
• mkdir obj sym dst
• xcodebuild install \-target ctfconvert -target ctfdump -target ctfmerge \
  ARCHS=x86_64 SRCROOT=$PWD OBJROOT=$PWD/obj \
  SYMROOT=$PWD/sym DSTROOT=$PWD/dst 
• sudo ditto \
  $PWD/dst/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain \
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
• cd ..
3. Install AvailabilityVersions
• tar zxf AvailabilityVersions-32.tar.gz
• cd AvailabilityVersions-32
• mkdir dst
• make install SRCROOT=$PWD DSTROOT=$PWD/dst
• sudo ditto \
  $PWD/dst/usr/local/libexec \
  $(xcrun -sdk macosx -show-sdk-path)/usr/local/libexec
• cd ..
4. Install libplatform headers
• tar zxf libplatform-161.tar.gz
• cd libplatform-161
• sudo mkdir -p \
  $(xcrun -sdk macosx -show-sdk-path)/usr/local/include/os/internal
• sudo ditto $PWD/private/os/internal \
  $(xcrun -sdk macosx -show-sdk-path)/usr/local/include/os/internal
• cd ..
5. Install XNU headers
• tar zxf xnu-4570.1.46.tar.gz
• cd xnu-4570.1.46
• make SDKROOT=macosx ARCH_CONFIGS=X86_64 installhdrs
• sudo ditto $PWD/BUILD/dst $(xcrun -sdk macosx -show-sdk-path)
• cd ..
6. Build firehose from libdispatch
• tar zxf libdispatch-913.1.6.tar.gz
• cd libdispatch-913.1.6
• mkdir obj sym dst
• xcodebuild install -sdk macosx -target libfirehose_kernel \
  SRCROOT=$PWD OBJROOT=$PWD/obj SYMROOT=$PWD/sym DSTROOT=$PWD/dst
• sudo ditto $PWD/dst/usr/local \
  $(xcrun -sdk macosx -show-sdk-path)/usr/local
• cd ..
7. Build XNU (checkout the README.md for more options!)
• cd xnu-4570.1.46
• make SDKROOT=macosx ARCH_CONFIGS=X86_64 KERNEL_CONFIGS=RELEASE

Check out the README.md file at the top of the XNU source tree for more options to the build system. Some common and useful options include: KERNEL_CONFIGS=DEVELOPMENT, BUILD_LTO=y  and LOGCOLORS=y.

Install and Run XNU

After this final step, you should have a new kernel built in $PWD/BUILD/obj/kernel. In order to run this kernel, you will need to install it, and rebuild the prelinked kernel cache. Installing a kernel could potentially render your system unbootable, so trying this out in a VM first is recommended. To install and run your kernel:

• cd xnu-4570.1.46
• sudo ditto $PWD/BUILD/obj/kernel /System/Library/Kernels/kernel
• sudo kextcache -invalidate /
  / locked; waiting for lock.
  Lock acquired; proceeding
  ...
• sudo reboot
  ...
• uname -a

If you build a different variant of XNU, you may need to ditto a different kernel name, e.g., kernel.development instead of just kernel.

Libsyscall

Additionally, if you would like to add system calls, you will need to build libsyscall_kernel. This library is built from the XNU source, but requires an additional dependency. The libsyscall library can be built and installed using the Makefile referenced above.

Warning: as of now, there are missing symbols in the resulting libsystem_kernel.dylib which will render your system unusable!

Here are the manual steps for building libsycall:

1. Follow steps 1-7 above to build XNU
2. Fixup libsyscall
• cd xnu-4570.1.46
• touch libsyscall/os/thread_self_restrict.h
• mkdir -p libsyscall/wrappers/skywalk/
• touch libsyscall/wrappers/skywalk/os_packet.c
• touch libsyscall/wrappers/skywalk/os_channel.c
• touch libsyscall/wrappers/skywalk/os_nexus.c
• touch libsyscall/wrappers/skywalk/cpu_in_cksum_gen.c
• touch libsyscall/wrappers/skywalk/cpu_copy_in_cksum_gen.c
• touch libsyscall/wrappers/skywalk/cpu_in_cksum.s
• touch libsyscall/wrappers/skywalk/cpu_copy_in_cksum.s
• sed -iE 's/ -lCrashReporterClient//' \$PWD/libsyscall/Libsyscall.xcconfig
3. Install XNU headers
• mkdir -p BUILD.hdrs/dst BUILD.hdrs/obj BUILD.hdrs/sym
• make installhdrs \
  SDKROOT=macosx \
  RC_ARCHS='x86_64 i386' \
  OBJROOT=$PWD/BUILD.hdrs/obj \
  SYMROOT=$PWD/BUILD.hdrs/sym \
  DSTROOT=$PWD/BUILD.hdrs/dst
• make installhdrs RC_ProjectName=Libsyscall \
  SDKROOT=macosx \
  RC_ARCHS='x86_64 i386' \
  OBJROOT=$PWD/BUILD.hdrs/obj \
  SYMROOT=$PWD/BUILD.hdrs/sym \
  DSTROOT=$PWD/BUILD.hdrs/dst
• sudo ditto $PWD/BUILD.hdrs/dst $(xcrun -sdk macosx -show-sdk-path)
4. Build libsyscall
• mkdir -p BUILD.sys/dst BUILD.sys/obj BUILD.sys/sym
• make install RC_ProjectName=Libsyscall \
  SDKROOT=macosx \
  RC_ARCHS='x86_64 i386' \
  OBJROOT=$PWD/BUILD.sys/obj \
  SYMROOT=$PWD/BUILD.sys/sym \
  DSTROOT=$PWD/BUILD.sys/dst
5. Install libsyscall
• sudo ditto $PWD/BUILD.sys/dst/usr/lib/system /usr/lib/system
• sudo update_dyld_shared_cache