Creating an Intel and OpenPower Heterogeneous Kubernetes Cluster

In this article let’s see how to manage a heterogeneous Kubernetes cluster consisting of Intel and OpenPower nodes.
Given below is my example setup along with the docker daemon command line used.

In my example setup, I don’t use any overlay network or tunneling like flannel, vxlan etc. While these are definitely required for very large scale setup, a small setup can be brought up without requiring these.

Assign labels to Kubernetes nodes
We’ll make use of ‘labels’ to provide hints to the scheduler so that it can schedule the Pods correctly. Labels are key/value pairs as shown below. We make use of ‘arch’ and ‘os’ label to describe the architecture and operating system of the Kubernetes node.
Latest version of Kubernetes automatically adds the architecture label  – ‘beta.kubernetes.io/arch=ppc64le‘ and os label beta.kubernetes.io/os. Consequently there won’t be any need to add arch and os label explicitly. For demonstration purposes, I’m adding the ‘os’ label explicitly. You can add any labels that you deem fit for your environment and use the same for scheduling the PODs.

# kubectl label nodes pkb-rhel71-1 os=rhel7
# kubectl label nodes pkb-rhel71-2 os=rhel7
# kubectl label nodes pkb-ub1404-1 os=ubuntu
# kubectl label nodes pkb-ub1404-2 os=ubuntu
# kubectl get nodes --show-labels
NAME             STATUS     AGE       LABELS
pkb-rhel71-1     Ready      11d       beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-1,misc=master,os=rhel7
pkb-rhel71-2     Ready      9d        beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-2,os=rhel7
pkb-ub1404-1     Ready      11d       beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-ub1404-1,os=ubuntu
pkb-ub1404-2     Ready      11d       beta.kubernetes.io/arch=amd64,kubernetes.io/hostname=pkb-ub1404-2,os=ubuntu

Please refer to my previous post on setting up Kubernetes cluster on OpenPower systems.

Provisioning Pods
We’ll make use of the nodeSelector field in the Pod specification which will then be used by the Kubernetes scheduler for provisioning. The following Pod specification is used to create mysql Pod on any OpenPower node.

# cat mysqlpod.yml
 apiVersion: "v1"
 kind: "Pod"
 metadata:
  name: "mysqlpod"
  labels:
    name: "db"
 spec:
     containers:
       - name: "mysql"
         image: "registry-rhel71.kube.com:5000/ppc64le/mysql"
         imagePullPolicy: "IfNotPresent"
         env:
            - name: MYSQL_USER
              value: test
            - name: MYSQL_PASSWORD
              value: test
            - name: MYSQL_ROOT_PASSWORD
              value: password
            - name: MYSQL_DB
              value: BucketList 
         ports:
           - containerPort: 3306
     nodeSelector:
         beta.kubernetes.io/arch: ppc64le

# kubectl create -f msqlpod.yml
# kubectl get pods -o wide
NAME       READY     STATUS   RESTARTS   AGE       NODE
mysqlpod   1/1       Running  0          3s        pkb-rhel71-1

Similarly the following Pod specification is used to create web Pod on any Intel node.

# cat webpod.yml
apiVersion: "v1"
 kind: "Pod"
 metadata:
  name: "webpod"
  labels:
    name: "web"
    uses: "db"
 spec:
     containers:
       - name: "web"
         image: "registry-rhel71.kube.com:5000/web"
         imagePullPolicy: "IfNotPresent"
         command: ["apache2ctl", "-D", "FOREGROUND"]
         env:
            - name: MYSQL_USER
              value: test
            - name: MYSQL_PASSWORD
              value: test
            - name: MYSQL_ROOT_PASSWORD
              value: password
            - name: MYSQL_DB
              value: BucketList
         ports:
           - containerPort: 80
 nodeSelector:
         beta.kubernetes.io/arch: amd64

# kubectl create -f webpod.yml
# kubectl get pods -o wide
NAME     READY     STATUS   RESTARTS   AGE       NODE
webpod   1/1       Running  0          3s        pkb-ub1404-2

As you can see by leveraging node labels and nodeSelector field in Pod specification we can effectively manage a heterogeneous Kubernetes cluster.