Build and Deploy an OpenPower based Kubernetes Cluster

Upcoming Ubuntu 16.04 code-named Xenial includes golang-1.6 for PowerPC LE (ppc64le) architecture. This makes it very easy to build Kubernetes from upstream source and use it to install and setup an OpenPower Kubernetes based cluster. In this article let’s see how to build Kubernetes from source and deploy a Kubernetes cluster on OpenPower servers running Ubuntu. Setting up of a Kubernetes cluster on OpenPower servers running RHEL LE have been covered in a separate article. Ofcourse you can have a heterogeneous cluster as well and I’ll write about it in a future blog post.

If you want to build Kubernetes on any Ubuntu version other than 16.04, then either you can build golang from source or install it from the 16.04 repository. For example this trick will allow installing golang-1.6 on Ubuntu Trusty (14.04).

# cat >/etc/apt/sources.list.d/golang.list<<EOF
deb http://ports.ubuntu.com/ubuntu-ports xenial main restricted
EOF
# apt-get update -y
# apt-get install -y golang-1.6
# go version
go version go1.6 linux/ppc64le

Building Kubernetes from source
As of this writing, upstream Kubernetes builds on Power with golang successfully.

[Update – Latest upstream Kubernetes builds are disabled on Power due to a golang issue. Check this github issue for more details – https://github.com/kubernetes/kubernetes/issues/25886]

Commit 4774e7146e0980b0ab9e0a8027e50254e7f6945d adds PowerPC LE (ppc64le) build support for Kubernetes.

# git clone https://github.com/kubernetes/kubernetes
# cd kubernetes
# git checkout -b ppc64le v1.3.6
# make

On successful build, the binaries will be available under ./_output/local/go/bin directory. By default most of the binaries that will be required for a kubernetes cluster – kube-apiservier, kube-controller-manager, kube-scheduler, kube-proxy are static binaries. Only kubelet and hyperkube binaries are not built as static by default. The kubelet and hyperkube dependencies are standard glibc dependencies. For example on my Ubuntu 14.04.03 LE host, the following is the dependency list:

# ldd /usr/bin/kubelet 
        linux-vdso64.so.1 =>  (0x00003fff966b0000)
        libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff96660000)
        libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff964a0000)
        /lib64/ld64.so.2 (0x000000003a930000)

hyperkube binary is an all-in-one binary encompassing kube-apiservier, kube-controller-manager, kube-scheduler, kube-proxy and kubelet.

Building Etcd from source
Kubernetes also requires etcd as it’s key-value distributed store. Building etcd from source on OpenPower systems using golang-1.6 is straight forward.

# git clone https://github.com/coreos/etcd
# cd etcd
# git checkout -b rel-2.2.5 v2.2.5
# ./build

On successful build, the etcd and etcdctl binaries will be available under the ./bin directory.

Build Kubernetes Infrastructure Container

Kubernetes infra container (gcr.io/google_containers/pause:0.8.0) is not available for Power from the google registry. You’ll be required to build it separately on Power and specify explicitly it’s location when configuring Kubernetes slaves on OpenPower systems.

[Update – Kubernetes infra container for Power architecture is available from google registry and all changes required for Power support is upstream. The below patch is no longer required ]

A small patch is required for building and creating the ‘pause’ container image on OpenPower system. The ‘pause’ source code is part of Kubernetes itself and can be found under kubernetes/build/pause directory.

# cd kubernetes
# curl https://github.com/bpradipt/kubernetes/commit/2425e00a81d259a66a31ca9de986164b6d5a651f.patch | git apply -
# cd build/pause
# make build
# docker images
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/ppc64le/pause        2.0                 7cba4204bba0        57 minutes ago      1.338 MB

Copy the ‘pause’ image in your private registry.

Once you have the required binaries and the ‘pause’ infra container, setting up and using a Kubernetes cluster on OpenPower servers is no different than on any other platform.

For completeness sake here is quick set of steps to bring up a Kubernetes cluster on OpenPower servers including setting up of private registry server.

Set up Private Registry Server
Kubernetes requires private registry to be TLS secured. In this example, a private registry server is setup on RHEL 7 LE system.
Create a directory to store the images. This could be created on any mount point on the server, backed by either local disk or external disk.
In my example setup /data/ is a separate partition on the disk where the registry storage will be created

# mkdir /data/registry_data

Create an HTTP access control file using ‘htpasswd’ command. The following command creates a file registry_passwd for user ‘kubeimguser. Replace the file name and user name as per your requirements. The option ‘-B’ is used for bcrypt encryption of passwords. This is required for docker private registry.

# htpasswd -Bc registry_passwd kubeimguser

Note that htpasswd is available as part of httpd-tools package on RHEL based systems and apache2-utils on Ubuntu based systems.

Create registry configuration file. Following is the config file used in this example setup.

version: 0.1
storage:
  filesystem:
    rootdirectory: /data/registry_data
  delete:
    enabled: true
http:
  addr: registry-rhel71.kube.com:5000
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/registry/registry_passwd

Create certificate for securing the registry using TLS.

# mkdir /certs/
# openssl req  -newkey rsa:4096 -nodes -sha256 -keyout /certs/domain.key  -x509 -days 365 -out /certs/domain.crt
Generating a 4096 bit RSA private key
..........................................................................................++
[snip]

Copy the certificate to all the Kubernetes nodes and install it under the path shown below:

# mkdir -p /etc/docker/certs.d/registry-rhel71.kube.com:5000/
# cp domain.crt /etc/docker/certs.d/registry-rhel71.kube.com:5000/ca.crt
# update-ca-trust
# service docker restart

Start the registry server as shown below:

# REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt  REGISTRY_HTTP_TLS_KEY=/certs/domain.key screen -dmS registry registry /etc/registry/config.yml

From any node in the cluster login to the private registry to get the registry authentication config file.

# docker login https://registry-rhel71.kube.com:5000
# cat /root/.docker/config.json 
{
	"auths": {
		"https://registry-rhel71.kube.com:5000": {
			"auth": "cHJhZGlwdGE6cHJhZGlwdGE=",
			"email": "test@test.com"
		}
}

Copy this config file (config.json) to all the nodes in the Kubernetes cluster under the same path (/root/.docker/config.json).
The cluster is now setup to use the private registry server.

Copy the images to private registry
In order to copy the above created ‘pause’ image to the private registry at registry-rhel71.powerkube.com:5000, perform the following steps:

# docker tag 7cba4204bba0  registry-rhel71.powerkube.com:5000/ppc64le/pause:2.0 
# docker push registry-rhel71.powerkube.com:5000/ppc64le/pause:2.0 

Example Cluster Setup
We’ll assume that the Kubernetes master and etcd will be running on the same OpenPower system (baremetal or VM) with IP 192.168.122.173.
Start etcd:

# screen -dmS etcd  etcd  --data-dir='/var/lib/etcd/default.etcd' --listen-client-urls='http://0.0.0.0:4001' --advertise-client-urls='http://0.0.0.0:4001, http://192.168.122.173:4001'

Start Kubernetes master processes:

# screen -dmS apiserver kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://192.168.122.173:4001 --address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
# screen -dmS controller-mgr kube-controller-manager --logtostderr=true --v=0 --master=http://192.168.122.173:8080
# screen -dmS scheduler kube-scheduler --logtostderr=true --v=0 --master=http://192.168.122.173:8080

Start kubelet and kube-proxy on the slaves:

# screen -dmS kubelet kubelet --logtostderr=true --v=0 --api-servers=http://192.168.122.173:8080 --address=0.0.0.0 --allow-privileged=false --pod-infra-container-image=registry-rhel71.powerkube.com:5000/ppc64le/pause:2.0
# screen -dmS proxy kube-proxy --logtostderr=true --v=0 --master=http://192.168.122.173:8080

This should setup your OpenPower Kubernetes Cluster. Here is a quick display from my example cluster setup:

# kubectl get nodes
NAME                           STATUS    AGE
ub1404-1-slave.powerkube.com   Ready     3d
ub1404-2-slave.powerkube.com   Ready     3d

Hope this will help you to leverage Kubernetes to deploy cloud-native applications on OpenPower systems.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...