Multi-architecture Docker Development and Test using QEMU

In this article we’ll look at using Qemu full-system emulation for setting up a multi-architecture Docker development environment. This kind of an environment is especially helpful when your primary development environment is Intel (x86_64), whereas you have a multi-architecture (Intel, Power, Arm) environment in production.

In one of my previous articles, we have looked at Qemu user mode emulation for the same purpose. You can read it here.

In the remainder of this article let’s at some of the available options for using Qemu full-system emulation.

Option-1 – Using Vagrant with libvirt provider

You can use Vagrant with libvirt provider and a PowerPC LE (ppc64le) box to run a ppc64le VM on Intel host. The host should have the qemu-system-ppc64le binary available. You can either install Docker on the VM via a provisioner (eg. shell) or you can create a box with Docker pre-installed. Instructions for creating a ppc64le Vagrant box from scratch can be found here.

Here is a sample Vagrantfile to create a ppc64le VM running on x86_64.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config| = "xenial-ppc64le"
  config.vm.synced_folder "/vagrant_data", "/vagrant_data", type: 'nfs'
  config.vm.hostname = "vm1-ppc"
  config.vm.provider :libvirt do |libvirt|
    #This is important. We need to use Qemu driver type for emulation
    libvirt.driver = "qemu"

    # The name of the server, where libvirtd is running. = "localhost"

    # If use ssh tunnel to connect to Libvirt.
     libvirt.connect_via_ssh = true

    # Libvirt storage pool name, where box image and instance snapshots will
    # be stored.
     libvirt.storage_pool_name = "default"

    # Set a prefix for the machines that's different than the project dir name.
    #libvirt.default_prefix = ''
  config.vm.define :vm_ppc64le do |vm_ppc64le|
    vm_ppc64le.vm.provider :libvirt do |domain|
      domain.memory = 2048
      domain.cpus = 1
      domain.video_type = "vga"
      domain.machine_arch = "ppc64le"
      domain.cpu_mode = "custom"
      domain.cpu_model = "POWER8"
  #Install docker for Ubuntu ppc64 from custom location
  config.vm.provision "shell", path: ""

The is a script to install Docker in the VM using ‘shell’ provisioner.
Here is the script that I use.

$ cat
RET=`dpkg -s docker-engine 2>/dev/null`
if [ $? != 0 ]
    echo deb [trusted=yes] xenial main > /etc/apt/sources.list.d/xenial-docker.list
    sudo apt-get update
    sudo DEBIAN_FRONTEND=noninteractive  apt-get install -y docker-engine

    #Add the vagrant user to docker group
    sudo adduser vagrant docker
    #Restart docker
    sudo service docker restart
    #kill all ssh connections from vagrant to the VM
    ps aux | grep 'sshd:' | awk '{print $2}' | xargs kill

Option-2- Using docker-machine

Docker Machine helps set up a Docker environment on virtual machines (VMs). Using Docker Machine one can install the Docker engine on VMs, and manage the VMs with docker-machine commands. Docker Machine supports provisioning VMs with docker engine installed on different types of environment (eg. different cloud providers like AWS, GCP etc).

We can use the Docker Machine tool to provision a Power (ppc64le) Docker environment on any Intel (x86_64) virtual machine, running either locally or in the cloud (AWS, GCP). Behind the scenes, we make use of qemu docker-machine driver plugin to provision the VM from a custom ppc64le ISO. The ppc64le VM runs on x86_64 by leveraging the full system emulation capability provided by Qemu TCG.  Currently only Ubuntu 16.04 is supported as the host to run ppc64le docker machine. The code is available here.

Please use the following steps to get started with installation and usage of the tool.


Intel Ubuntu 16.04 VM (min 2 vCPU, 4GB RAM, 20GB disk)

Run the following command on Intel Ubuntu 16.04 VM to setup the Power Docker environment.

$ wget && chmod +x ./ && ./ -s

The overall installation takes around 15-20 min, including the time spent in downloading the ISO image.

Exit the session and log in again to the Intel VM.

Run the following command to create the machine

$ docker-machine create -d qemu --qemu-boot2docker-url=/home/ubuntu/.docker/machine/boot2docker.iso --qemu-memory 2048 --qemu-cache-mode none --qemu-arch ppc64le vm1 

This should take around 4-5 min approximately.

Now you can use docker-machine commands to interact with the machine and start building and testing your docker images for Power.

If you are new to docker-machine here are some handy commands that will get you started.

1. List machines

$ docker-machine ls
NAME   ACTIVE   DRIVER   STATE     URL                                   SWARM   DOCKER        ERRORS
vm1    -        qemu     Running   tcp://                     v17.03.1-ce

2. List environment variables for a specific machine

$ docker-machine env vm1
export DOCKER_HOST="tcp://"
export DOCKER_CERT_PATH="/home/ubuntu/.docker/machine/machines/vm1"
# Run this command to configure your shell:
# eval $(docker-machine env vm1)

3. Configure shell for enabling docker client to access a specific machine

$ eval $(docker-machine env vm1)

$ docker version
 Version:      17.05.0-ce-rc3
 API version:  1.27 (downgraded from 1.29)
 Go version:   go1.7.5
 Git commit:   90d35ab
 Built:        Tue May  2 17:48:51 2017
 OS/Arch:      linux/amd64

 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Jun  2 17:34:21 2017
 OS/Arch:      linux/ppc64le
 Experimental: false

4. SSH to the machine

$ docker-machine ssh vm1

5. Run a ppc64le docker image

$ docker run -itd ppc64le/httpd

Since this is using emulation, performance is slower when compared to native execution. However, it should be able to give you an environment to build and verify a ppc64le docker image without depending on the availability of a native ppc64le environment. We expect the performance to be better with availability of multi-threaded TCG for Power in the coming future.

Do share your feedback and suggestions in the comments section.

[Note – The docker-machine contents first appeared here ]

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...