In this post I will explain how to build and push images for different platforms using a single Dockerfile and how to create a multi-platform manifest that allows Docker client to automatically pull and run the correct image for the platform.

In September 2017, Docker updated their official images to make them multi-platform aware. Docker clients will pull and run the correct Docker image for your platform whether that is x86-64 Linux, ARM or any other system with Docker. As of February 2018 the Docker client has a manifest tool that allow you to create multi-platform images yourself. The manifest tool is currently an experimental CLI feature so to be able to use it you have to enable experiment features for your Docker client. If you don’t want to do this you can also use manifest-tool as an alternative. This post will assume the Docker manifest tool.

The problem

As most of my Dockerfile‘s are quite simple and I don’t want to maintain Dockerfile‘s for every platform I was looking for a way to build my images from a single source. Another problem I had was that I wanted to be able to cross build my arm images with a gitlab-runner running on an x86_64 host.

Approach

There is an excellent blog post that explains how to run and build ARM Docker containers on a x86 host using QEMU. The post also explains how to reclaim the extra disk space used to create these kind of images, this is currently an experimental function of the Docker daemon, I won’t get into this for now.

In short you will need to install QEMU user mode and add the qemu-arm-static interpreter inside the container you want to cross build. By using a build argument you can enable cross-builds on a Dockerfile.

First install qemu:

apt-get install qemu-user qemu-user-static

Consider the following Dockerfile to create a nginx container, we will assume you are using a x86 host for building:

ARG BASE_IMAGE=debian:stretch-slim
FROM $BASE_IMAGE

COPY qemu-arm-static /usr/bin

RUN apt-get update;\
    apt-get install -y nginx;\
    echo "\ndaemon off;" >> /etc/nginx/nginx.conf

RUN rm /usr/bin/qemu-arm-static

CMD ["nginx"]

Before building the image you will need to copy qemu-arm-static to the folder your are building your image from. When you docker build this Dockerfile you will create an image for the x86 platform that runs nginx:

docker build -t nginx/linux-x86_64:latest

When we now add the arm32v7 Debian base image we can cross build an image that supports the arm platform.

docker build --build-arg BASE_IMAGE=arm32v7/debian:stretch-slim -t nginx/linux-armv7:latest

You can even run this image on your host by re-inserting qemu-arm-static:

docker run -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static --rm -ti nginx/linux-armv7:latest

Note that we copy the qemu-arm-static binary into the image without actually needing it. This will of course add an extra build step and grow our image. In the future we will be able to reclaim the extra space using the --squash option of docker build but for now I accept this overhead in favour of having a single Dockerfile.

Pushing the images to the server

Before we start creating the manifest let us first push these images to a Docker repository:

docker push nginx/linux-x86_64:latest
docker push nginx/linux-armv7:latest

Creating the manifest

The next step is now to create a manifest that allows the docker client to automatically select the right image for the host platform. First you will need to enable the experimental CLI features by adding "experimental": "enabled" to your Docker configuration which is located in ~/.docker/config.json. Here is mine:

{
    "experimental": "enabled"
}

Now that the experimental features are enabled we can create the manifest with the following command:

docker manifest create nginx:latest nginx/linux-x86_64:latest nginx/linux-armv7:latest

This command creates the manifest and adds the manifests of the 2 containers that we want to add. The next step is to annotate the manifest with information about target platforms and push it to the repository:

docker manifest annotate --os linux --arch amd64 nginx:latest nginx/linux-x86_64:latest
docker manifest annotate --os linux --arch amd65 nginx:latest nginx/linux/armv7:latest
docker manifest push nginx:latest

That it. Now you will be able to use the nginx:latest tag to retrieve the right image whether you hosts run on linux-x86_64 or armv7.

Continues integration using GitLab runners

It is tedious work to manually to all of this every time you need to update your container. I use GitLab for version control, issue management en CI/CD. Gitlab runners can be used to automate all sorts of build, test and deploy tasks, including building docker containers.

Once setup the Gitlab runner correctly, to enable docker-in-docker builds, you can use the .gitlab-ci.yml script below to deploy intermediate and release builds for you docker containers. The build phase needs access to qemu. I choose to download and inject it but you can also add the binary to your git repository. The script first builds and pushes the platform specific containers and finally it will create and push the platform agnostic manifest. This script pushes the images to the Gitlab intern registry but it is also possible to push to another registry including the Docker Hub.

image: docker:stable
services:
- docker:dind

stages:
- build
- manifest

variables:
  CONTAINER_IMAGE: $CI_REGISTRY_IMAGE

# Before building we need to enable the experimental features and login into the registry.
# For this I've included a temlate configuration in my project.
before_script:
  - docker info
  - mkdir -p /root/.docker || true
  - cp etc/docker-config.json /root/.docker/config.json
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

build64:
  stage: build
  script:
    - docker build
      -t $CONTAINER_IMAGE/linux-amd64:latest
      --build-arg BASE_IMAGE=debian:stretch-slim .
    - docker push $CONTAINER_IMAGE/linux-amd64:latest

build-arm:
  stage: build
    # Download qemu static and make it executable so it can be used during the docker build phase
    - wget https://github.com/multiarch/qemu-user-static/releases/download/v2.12.0/qemu-arm-static -O qemu-arm-static
    - chmod 554 qemu-arm-static
    - docker build
      -t $CONTAINER_IMAGE/linux-arm:latest
      --build-arg BASE_IMAGE=arm32v7/debian:stretch-slim .
    - docker push $CONTAINER_IMAGE/linux-arm:latest

manifest:
  stage: manifest
  script:
    # first make sure we have the containers we use in the manifest
    - docker pull $CONTAINER_IMAGE/linux-amd64:latest
    - docker pull $CONTAINER_IMAGE/linux-arm:latest
    - docker manifest create $CONTAINER_IMAGE:latest
        $CONTAINER_IMAGE/linux-amd64:latest
        $CONTAINER_IMAGE/linux-arm:latest
    - docker manifest annotate --os linux --arch amd64
        $CONTAINER_IMAGE:latest
        $CONTAINER_IMAGE/linux-amd64:latest
    - docker manifest annotate --os linux --arch arm --variant v6
        $CONTAINER_IMAGE:latest
        $CONTAINER_IMAGE/linux-arm:latest
    # the login to the gitlab registry sometimes times out so make sure
    # we are logged on before pushing the manifest
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker manifest push $CONTAINER_IMAGE:latest

Using the script above will automatically build and push docker containers and manifest for your project to the Gitlab internal registry. Of course you can improve the behavior of the script to build images only whenever a release version gets tagged in Git and to use the Git tag as a label for you docker container if you want to. That may be a topic for another post…