A container orchestration software is required if your environment consists of multiple hosts running docker containers. There are numerous options available today – Docker Swarm, Mesos/Marathon, Kubernetes among others. This article will help you to setup a Kubernetes cluster on OpenPower systems (eg. Tyan) running RHEL LE (little endian). The same instructions also applies to other scale-out Power servers from IBM.
Kubernetes is an opensource orchestration engine for docker containers and works on the master-slave concept. Following are the major components of a Kubernetes cluster:
- Master: this is the cluster manager which oversees one or more nodes (minions).
- Node or Minion or Slave: these are cluster members and responsible for starting containers.
- Pod: this is the basic unit of operation in Kubernetes. It represents a group of one or more containers constituting an application (or part) that runs on a slave (minion).
Kubernetes includes lot many other concepts which are necessary for a real production deployment. While going into the details of all of them is beyond the scope of this article, nonetheless a good reference is the following documentation – http://kubernetes.io/v1.1/docs/user-guide/walkthrough/k8s201.html.
Following are the instructions which should help you with Kubernetes setup. Special thanks to my colleague Peeyush Gupta for co-authoring this writeup.
Installation and Setup of Kubernetes
While you can build Kubernetes from source on Power platform however for ease of use, packages for RHEL LE are provided on an as-is basis from the Unicamp repository.
Add Unicamp Package Repository for RHEL
Ensure the following repositories are added to all the systems that are going to be part of the Kubernetes cluster
# cat > /etc/yum.repos.d/unicamp-docker.repo <<EOF [unicamp-docker] name=Unicamp Repo for Docker Packages baseurl=http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/ enabled=1 gpgcheck=0 EOF # cat > /etc/yum.repos.d/unicamp-misc.repo <<EOF [unicamp-misc] name=Unicamp Repo for Misc Packages baseurl=http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/misc_ppc64el/ enabled=1 gpgcheck=0 EOF
Add Unicamp Package Repository for Advance Toolchain
You’ll need a Go compiler to build docker image for Kubernetes infrastructure container. Additionally you’ll need the compiler if you would like to build Kubernetes from source
# cat > /etc/yum.repos.d/at9.repo <<EOF [at9] name=Advance Toolchain Unicamp FTP baseurl=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7 enabled=1 gpgcheck=1 gpgkey=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7/gpg-pubkey-6976a827-5164221b EOF
Install Advance toolchain
# yum install -y advance-toolchain-at9.0-runtime \ advance-toolchain-at9.0-devel \ advance-toolchain-at9.0-perf \ advance-toolchain-at9.0-mcore-libs # echo "export PATH=/opt/at9.0/bin:/opt/at9.0/sbin:$PATH" >> /etc/profile.d/at9.sh # source /etc/profile.d/at9.sh # /opt/at9.0/sbin/ldconfig
Building Infra container image – pause
[Note: This step is no longer required since the changes to build the ‘pause’ infra binary are available upstream. Additionally ‘pause’ infra container image for PowerPC LE is available from Google registry – gcr.io/google_containers/pause-ppc64le]
This is a special type of container created by default for every POD. It just sleeps and is used to provide networking connectivity to other containers in the POD. By default Kubernetes downloads it from ‘gcr.io/google_containers/pause’. However since no ‘pause’ image exists for Power platforms yet in Google registry, the only way possible is to build the ‘pause’ container image on Power and use the same.
A slightly modified version of the ‘pause’ code allowing it to be built on Power platforms can be downloaded from Peeyush’s github tree. The instructions are described below:
# git clone https://github.com/Pensu/pause.git # cd pause # make
After make, run “docker images” to see the pod infra container image in the list.
[root@localhost pause]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE pause 0.8.0 f149b36756ff About an hour ago 7.994 MB
Tag the image as appropriate and either push it to your local registry or to public docker registry. The full image name needs to be updated in the /etc/kubernetes/kubelet file for all the nodes (minions).
Installation and Setup of Kubernetes Master
Install the required packages
# yum install kubernetes-client kubernetes-master etcd
Open Network Ports
By default kubernetes apiserver listens on port 8080 for kubelets. Ensure that it is not blocked by local firewall. If using firewalld, then the following command can be used to open a TCP port for the ‘public’ zone
# firewall-cmd --zone=public --add-port=8080/tcp --permanent # firewall-cmd --reload
Additional the etcd server listens on port 2379 by default. Use the following instructions to open the respective port.
# firewall-cmd --zone=public --add-port=2379/tcp --permanent # firewall-cmd --reload
Configure Kubernetes Master
For the remainder of the configuration we’ll assume that the Kubernetes master has the following IP – 192.168.122.76, and the Kubernetes node has the following IP – 192.168.122.236.
Modify the file /etc/kubernetes/config according to the environment. Based on the above info, the modified file will look the following:
# logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" # journal message level, 0 is debug KUBE_LOG_LEVEL="--v=0" # Should this cluster be allowed to run privileged docker containers KUBE_ALLOW_PRIV="--allow-privileged=false" # How the controller-manager, scheduler, and proxy find the apiserver KUBE_MASTER="--master=http://192.168.122.76:8080"
Modify /etc/kubernetes/apiserver according to the environment. Based on the above info, the modified file will look the following:
# The address on the local server to listen to. KUBE_API_ADDRESS="--address=0.0.0.0" # The port on the local server to listen on. # KUBE_API_PORT="--port=8080” # Port minions listen on # KUBELET_PORT="--kubelet-port=10250" # Comma separated list of nodes in the etcd cluster KUBE_ETCD_SERVERS="--etcd-servers=http://192.168.122.76:2379" # Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" # default admission control policies KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota" # Add your own! KUBE_API_ARGS=""
Modify the following two parameters in /etc/etcd/etcd.conf as shown below:
Start the services
# for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done
Installation and Setup of Kubernetes Node (Minion)
Install the required packages
# yum install docker-io kubernetes-client kubernetes-node
Configure Kubernetes Node
Modify /etc/kubernetes/ kubelet according to the environment. Based on the above info, the modified file will look the following:
# kubernetes kubelet (minion) config # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces) KUBELET_ADDRESS="--address=0.0.0.0" # The port for the info server to serve on # KUBELET_PORT="--port=10250" # You may leave this blank to use the actual hostname KUBELET_HOSTNAME=" " # location of the api-server KUBELET_API_SERVER="--api-servers=http://192.168.122.76:8080" # Add your own! KUBELET_ARGS="--pod-infra-container-image=<IMAGE_WITH_REGISTRY_LOCATION>"
Ensure to update the “pod-infra-container-image” parameter with full details of the infra container image.
Start the services
# for SERVICES in kube-proxy kubelet docker; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done
Verifying the setup
Log in to the master and run “kubectl get nodes” to check the available nodes.
[root@localhost ~]# kubectl get nodes NAME LABELS STATUS AGE fed-node kubernetes.io/hostname=fed-node Ready 1h
Hope this helps you to use Kubernetes to manage multiple OpenPower based docker hosts.