Creating an Intel and OpenPower Heterogeneous Kubernetes Cluster

In this article let’s see how to manage a heterogeneous Kubernetes cluster consisting of Intel and OpenPower nodes.
Given below is my example setup along with the docker daemon command line used.

Node Docker Daemon Start Options
Host1 (RHEL 7.1 LE) docker daemon -b br0 --fixed-cidr=172.16.0.32/27
Host2 (RHEL 7.1 LE) docker daemon -b br0 --fixed-cidr=172.16.127.32/27
Host3 (Ubuntu 14.04.03 LE) docker daemon -b br0 --fixed-cidr=172.16.0.64/27
Host4 (Ubuntu 14.04.03 x86_64) docker daemon -b br0 --fixed-cidr=172.16.127.64/27

In my example setup, I don’t use any overlay network or tunneling like flannel, vxlan etc. While these are definitely required for very large scale setup, a small setup can be brought up without requiring these.

Assign labels to Kubernetes nodes
We’ll make use of ‘labels’ to provide hints to the scheduler so that it can schedule the Pods correctly. Labels are key/value pairs as shown below. We make use of ‘arch’ and ‘os’ label to describe the architecture and operating system of the Kubernetes node.

# kubectl label nodes pkb-rhel71-1 arch=ppc64le os=rhel7
# kubectl label nodes pkb-rhel71-2 arch=ppc64le os=rhel7
# kubectl label nodes pkb-ub1404-1 arch=ppc64le os=ubuntu
# kubectl label nodes pkb-ub1404-2 arch=x86_64 os=ubuntu
# kubectl get nodes --show-labels
NAME             STATUS     AGE       LABELS
pkb-rhel71-1     Ready      11d       arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-1,misc=master,os=rhel7
pkb-rhel71-2     Ready      9d        arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-2,os=rhel7
pkb-ub1404-1     Ready      11d       arch=ppc64le,kubernetes.io/hostname=pkb-ub1404-1,os=ubuntu
pkb-ub1404-2     Ready      11d       arch=x86_64,kubernetes.io/hostname=pkb-ub1404-2,os=ubuntu

Please refer to my previous post on setting up Kubernetes cluster on OpenPower systems.

Provisioning Pods
We’ll make use of the nodeSelector field in the Pod specification which will then be used by the Kubernetes scheduler for provisioning. The following Pod specification is used to create mysql Pod on any OpenPower node.

# cat mysqlpod.yml
 apiVersion: "v1"
 kind: "Pod"
 metadata:
  name: "mysqlpod"
  labels:
    name: "db"
 spec:
     containers:
       - name: "mysql"
         image: "registry-rhel71.kube.com:5000/ppc64le/mysql"
         imagePullPolicy: "IfNotPresent"
         env:
            - name: MYSQL_USER
              value: test
            - name: MYSQL_PASSWORD
              value: test
            - name: MYSQL_ROOT_PASSWORD
              value: password
            - name: MYSQL_DB
              value: BucketList 
         ports:
           - containerPort: 3306
     nodeSelector:
         arch: ppc64le
# kubectl create -f msqlpod.yml
# kubectl get pods -o wide
NAME       READY     STATUS   RESTARTS   AGE       NODE
mysqlpod   1/1       Running  0          3s        pkb-rhel71-1

Similarly the following Pod specification is used to create web Pod on any Intel node.

# cat webpod.yml
apiVersion: "v1"
 kind: "Pod"
 metadata:
  name: "webpod"
  labels:
    name: "web"
    uses: "db"
 spec:
     containers:
       - name: "web"
         image: "registry-rhel71.kube.com:5000/web"
         imagePullPolicy: "IfNotPresent"
         command: ["apache2ctl", "-D", "FOREGROUND"]
         env:
            - name: MYSQL_USER
              value: test
            - name: MYSQL_PASSWORD
              value: test
            - name: MYSQL_ROOT_PASSWORD
              value: password
            - name: MYSQL_DB
              value: BucketList
         ports:
           - containerPort: 80
 nodeSelector:
         arch: x86_64
# kubectl create -f webpod.yml
# kubectl get pods -o wide
NAME     READY     STATUS   RESTARTS   AGE       NODE
webpod   1/1       Running  0          3s        pkb-ub1404-2

As you can see by leveraging node labels and nodeSelector field in Pod specification we can effectively manage a heterogeneous Kubernetes cluster.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...