In this article let’s see how to manage a heterogeneous Kubernetes cluster consisting of Intel and OpenPower nodes.
Given below is my example setup along with the docker daemon command line used.
Node | Docker Daemon Start Options |
---|---|
Host1 (RHEL 7.1 LE) | docker daemon -b br0 –fixed-cidr=172.16.0.32/27 |
Host2 (RHEL 7.1 LE) | docker daemon -b br0 –fixed-cidr=172.16.127.32/27 |
Host3 (Ubuntu 14.04.03 LE) | docker daemon -b br0 –fixed-cidr=172.16.0.64/27 |
Host4 (Ubuntu 14.04.03 x86_64) | docker daemon -b br0 –fixed-cidr=172.16.127.64/27 |
In my example setup, I don’t use any overlay network or tunneling like flannel, vxlan etc. While these are definitely required for very large scale setup, a small setup can be brought up without requiring these.
Assign labels to Kubernetes nodes
We’ll make use of ‘labels’ to provide hints to the scheduler so that it can schedule the Pods correctly. Labels are key/value pairs as shown below. We make use of ‘arch’ and ‘os’ label to describe the architecture and operating system of the Kubernetes node.
Latest version of Kubernetes automatically adds the architecture label – ‘beta.kubernetes.io/arch=ppc64le‘ and os label beta.kubernetes.io/os. Consequently there won’t be any need to add arch and os label explicitly. For demonstration purposes, I’m adding the ‘os’ label explicitly. You can add any labels that you deem fit for your environment and use the same for scheduling the PODs.
# kubectl label nodes pkb-rhel71-1 os=rhel7 # kubectl label nodes pkb-rhel71-2 os=rhel7 # kubectl label nodes pkb-ub1404-1 os=ubuntu # kubectl label nodes pkb-ub1404-2 os=ubuntu # kubectl get nodes --show-labels NAME STATUS AGE LABELS pkb-rhel71-1 Ready 11d beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-1,misc=master,os=rhel7 pkb-rhel71-2 Ready 9d beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-rhel71-2,os=rhel7 pkb-ub1404-1 Ready 11d beta.kubernetes.io/arch=ppc64le,kubernetes.io/hostname=pkb-ub1404-1,os=ubuntu pkb-ub1404-2 Ready 11d beta.kubernetes.io/arch=amd64,kubernetes.io/hostname=pkb-ub1404-2,os=ubuntu
Please refer to my previous post on setting up Kubernetes cluster on OpenPower systems.
Provisioning Pods
We’ll make use of the nodeSelector field in the Pod specification which will then be used by the Kubernetes scheduler for provisioning. The following Pod specification is used to create mysql Pod on any OpenPower node.
# cat mysqlpod.yml apiVersion: "v1" kind: "Pod" metadata: name: "mysqlpod" labels: name: "db" spec: containers: - name: "mysql" image: "registry-rhel71.kube.com:5000/ppc64le/mysql" imagePullPolicy: "IfNotPresent" env: - name: MYSQL_USER value: test - name: MYSQL_PASSWORD value: test - name: MYSQL_ROOT_PASSWORD value: password - name: MYSQL_DB value: BucketList ports: - containerPort: 3306 nodeSelector: beta.kubernetes.io/arch: ppc64le
# kubectl create -f msqlpod.yml # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE NODE mysqlpod 1/1 Running 0 3s pkb-rhel71-1
Similarly the following Pod specification is used to create web Pod on any Intel node.
# cat webpod.yml apiVersion: "v1" kind: "Pod" metadata: name: "webpod" labels: name: "web" uses: "db" spec: containers: - name: "web" image: "registry-rhel71.kube.com:5000/web" imagePullPolicy: "IfNotPresent" command: ["apache2ctl", "-D", "FOREGROUND"] env: - name: MYSQL_USER value: test - name: MYSQL_PASSWORD value: test - name: MYSQL_ROOT_PASSWORD value: password - name: MYSQL_DB value: BucketList ports: - containerPort: 80 nodeSelector: beta.kubernetes.io/arch: amd64
# kubectl create -f webpod.yml # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE NODE webpod 1/1 Running 0 3s pkb-ub1404-2
As you can see by leveraging node labels and nodeSelector field in Pod specification we can effectively manage a heterogeneous Kubernetes cluster.