Automated Build Using CircleCI

Difficulty level
Medium

This tutorial will show you how to set up an automated build environment using CircleCI, which will automatically build your project on every push to a repository, and make the resulting files available on the web. In the tutorial it is assumed that your kernel / operating system project is hosted at GitHub, from where it can be cloned and built using a freestanding GCC Cross-Compiler toolchain.

Setting up your project on CircleCI

First you will need to sign up at CircleCI. Navigate to their homepage and sign in. You will then have the possibility to connect with your GitHub account. GitHub will ask you whether you would like to grant CircleCI access to your account. Once you have confirmed, you can enter your CircleCI settings. Choose a project to build to enable automated builds whenever you push to that project.

Add a build script for the toolchain

The first step towards building your project should be building the toolchain, and thus in particular a GCC Cross-Compiler. The most convenient way to do this is to create a small script that will download the sources for binutils and gcc (and optionally additional sources, such as newlib or gdb), configure them for your chosen target and build the tools. The following script is an example, which builds a toolchain suitable for using a subset of C++ (only the freestanding part):

#!/bin/sh
set -e

# Set the versions we will be using.
binutils_version="2.27"
gcc_version="6.3.0"
newlib_version="2.4.0"

# This script expects the target triplet (e.g. i786-pc-elf) as command line argument.
target=$1

# The tools will be installed in ~/cross/$target.
prefix=~/cross/$target

# First check whether the toolchain was already built on a previous run of this script.
if [ ! -d $prefix ]
then
	mkdir -p /tmp/toolchain
	cd /tmp/toolchain

	# Download gcc sources if they are not yet downloaded.
	if [ ! -f gcc-$gcc_version.tar.bz2 ]
	then
		wget -c -O gcc-$gcc_version.tar.bz2 ftp://ftp.gnu.org/gnu/gcc/gcc-$gcc_version/gcc-$gcc_version.tar.bz2
		tar -xf gcc-$gcc_version.tar.bz2
	fi

	# Download binutils sources if they are not yet downloaded.
	if [ ! -f binutils-$binutils_version.tar.bz2 ]
	then
		wget -c -O binutils-$binutils_version.tar.bz2 ftp://ftp.gnu.org/gnu/binutils/binutils-$binutils_version.tar.bz2
		tar -xf binutils-$binutils_version.tar.bz2
	fi

	# Optional: download newlib sources if they are not yet downloaded.
	if [ ! -f newlib-$newlib_version.tar.gz ]
	then
		wget -c -O newlib-$newlib_version.tar.gz ftp://sources.redhat.com/pub/newlib/newlib-$newlib_version.tar.gz
		tar -xf newlib-$newlib_version.tar.gz
	fi

	# Create build paths.
	mkdir -p /tmp/toolchain/build-binutils
	mkdir -p /tmp/toolchain/build-gcc
	mkdir -p /tmp/toolchain/build-newlib

	# Build binutils.
	cd /tmp/toolchain/build-binutils
	sudo rm -rf *
	/tmp/toolchain/binutils-$binutils_version/configure --target=$target --prefix=$prefix --disable-nls 2>&1
	make all 2>&1
	make install 2>&1
	sudo rm -rf *

	# Build gcc and libgcc.
	cd /tmp/toolchain/build-gcc
	/tmp/toolchain/gcc-$gcc_version/configure --target=$target --prefix=$prefix --disable-nls --enable-languages=c,c++ --enable-libstdcxx --without-headers 2>&1
	make all-gcc 2>&1
	make install-gcc 2>&1
	make all-target-libgcc 2>&1
	make install-target-libgcc 2>&1

	# Make sure that our cross compiler will be found by creating links.
	# Alternative: Add the $prefix/bin directory to your $PATH.
	sudo ln -s -f $prefix/bin/* /usr/local/bin/

	# Optional: Build newlib. This is necessary only for the next, also optional build step.
	cd /tmp/toolchain/build-newlib
	sudo rm -rf *
	/tmp/toolchain/newlib-$newlib_version/configure --target=$target --prefix=$prefix 2>&1
	make all 2>&1
	make install 2>&1
	sudo rm -rf *

	# Optional: Build libstdc++. This is done in order to install the freestanding headers for using the C++11, C++14, C++17 standards.
	cd /tmp/toolchain/build-gcc
	/tmp/toolchain/gcc-$gcc_version/configure --target=$target --prefix=$prefix --disable-nls --enable-languages=c,c++ --enable-libstdcxx --without-headers --with-newlib 2>&1
	make all-target-libstdc++-v3 2>&1
	make install-target-libstdc++-v3 2>&1
	sudo rm -rf *
fi

# Also if the cross compiler has not been freshly build, link it so that it will be found.
sudo ln -s -f $prefix/bin/* /usr/local/bin/

Note that this is only an example script. You will most likely have to modify it according to your needs. Also you should test locally whether it actually works and generates the compiler you will use later. Once you have done this, add it to your source repository. Here we will assume that you have created a directory called ci in the root of your repository, and placed your script as a file named toolchain.sh in this folder.

Set up the build

Now we need to tell CircleCI which actions it should perform when you push to your repository. By default, CircleCI uses a virtual machine running Ubuntu 14.04 for your build. This is configured using a file called circle.yml in the root of your repository. The following example explains the sections in this file.

dependencies:
    pre:
        # Install a few packages which will be needed for building the cross compiler.
        - sudo apt-get update
        - sudo apt-get install grub-pc
        - sudo apt-get install libgmp3-dev
        - sudo apt-get install libmpfr-dev
        - sudo apt-get install libmpc-dev
        - sudo apt-get install texinfo
        - sudo apt-get install libisl-dev
        - sudo apt-get install libcloog-isl-dev

        # Build the cross compiler for your chosen target (replace i786-pc-elf if necessary).
        - bash ci/toolchain.sh i786-pc-elf

    cache_directories:
        # Add the folder with our cross compiler to the cache. This way it will be kept between builds.
        - ~/cross

compile:
    override:
        # Place here whatever commands are necessary to build your project.
        - bash your_build_script.sh

        # Copy any files you want to keep to the folder $CIRCLE_ARTIFACTS.
        - cp my_kernel my_image.iso $CIRCLE_ARTIFACTS/

test:
    override:
        # Place here any automated test you would like to run after the build (unit tests, boot up a VM and collect output...)
        - bash your_test_suite.sh

        # You must place something here, even it it's just testing whether the output exists.
        - test -f $CIRCLE_ARTIFACTS/my_kernel
        - test -f $CIRCLE_ARTIFACTS/my_image.iso

Ones you have created a custom file of this type, save it as circle.yml in the root of your repository, commit and push. Now CircleCI should start building your cross compiler, go on with building your project, collect the output files, check for success and inform you whether the build succeeded. The next time you push, all cached directories (and thus your cross compiler) will be restored before the build starts (and so your cross compiler will not be built again on every push).

Publish outputs on the web

Both on CircleCI and on the commits page or your GitHub project you should now see whether your build was successful. If you would also like to show the build status on your project homepage, you can use a link of one of these forms:

https://circleci.com/gh/<your_user_name>/<your_project_name>.svg?style=svg
https://circleci.com/gh/<your_user_name>/<your_project_name>.svg?style=shield

You can also create links to the files you save in the $CIRCLE_ARTIFACTS folder in your build. Here the link should look like this:

https://circleci.com/api/v1/project/<your_user_name>/<your_project_name>/latest/artifacts/0/$CIRCLE_ARTIFACTS/<your_file_name>

This will always link to the latest generated files.

Links

• Configuring CircleCI
• CircleCI build status badges
• CircleCI build artifacts