Howto modify a Kubeadm based Kubernetes Environment

Kubernetes provides multiple options for setting up a development environment. A very simple method is to use minikube. However it’s only a single node solution. Hence for cases where you would like to simulate real deployments or understand multi-node networking or play with scheduler options minikube is not suitable. Starting Kubernetes version 1.4 a new tool called kubeadm is made available which makes multi-node kubernetes cluster setup for dev and test a breeze.

If you have used the latest docker engine swarm mode you will find the kubeadm setup instructions very similar.

Kubernetes master server runs ‘kubeadm init [options]’ to initialize the cluster. Kubernetes slaves runs ‘kubeadm join [options]’ to become part of the cluster. This is very similar to ‘docker swarm init [options]’ and ‘docker swarm join [options]’.

Here is what swarm cluster setup instructions looks like.

docker swarm init --advertise-addr 192.168.122.173
Swarm initialized: current node (0gqevumsltdkz8bo8a4ybprii) is now a manager.

To add a worker to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-14dpd920dwirq5k6bqm3h4x1lwy4m1fiopdiw3ykorlxkio9aq-do6misrpzzx77r4jwd4qfvu02 \
    192.168.122.173:2377

If you are interested in the inner working of kubeadm, the following link is a good start – http://kubernetes.io/docs/admin/kubeadm/

In this article we’ll see how to modify a Kubernetes cluster configuration that has been setup using kubeadm.

First, we’ll setup a cluster. The example environment consists of two Ubuntu 16.04 VMs and follows the steps outlined in the user-guide – http://kubernetes.io/docs/getting-started-guides/kubeadm/

Install pre-requisite packages

Perform the following steps on both the Ubuntu VMs.

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
# apt-get update
# # Install docker if you don't have it already.
# apt-get install -y docker.io
# apt-get install -y kubelet kubeadm kubectl kubernetes-cni

After installing the required packages, setting up the cluster is simply about running the following commands on master and slaves.

Kubernetes master host

Step-1: Initialise the control-plane

Kubernetes control-plane consists of the following components:
• Kubernetes api-server (kube-apiserver)
• Kubernetes scheduler (kube-scheduler)
• Kubernetes controller-manager (kube-controller-manager)
• Etcd (etcd)

# kubeadm init --api-advertise-addresses 192.168.33.10

[snip]
Kubernetes master initialised successfully!

You can now join any number of machines by running the following on each node:

kubeadm join --token=f417d7.c3e4fe8139088285 192.168.33.10

Step-2: Setup networking for communication among pods.

I prefer to use either Calico or Flannel. You can refer to the following doc for networking alternatives – http://kubernetes.io/docs/admin/addons/

Setting up a pod network is must before you can start any application deployment. Perform the following steps for setting up Calico.

# kubectl apply –f http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml

Verify if pod network is working by checking if kube-dns pod is in ‘Running’ state from the output of ‘kubetl get pods –all-namespaces’

The above two steps completes the process on the master host.

Kubernetes slave host

Step-1: Setup slave

Execute the relevant command that was displayed in the output of the ‘kubeadm init’ when initialising the cluster. This will make the slave part of the cluster.

# kubeadm join --token=f417d7.c3e4fe8139088285 192.168.33.10
[snip]
Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join

Executing ‘kubectl get nodes’ on the master displays the following:

# kubectl get nodes
NAME      STATUS    AGE
kubevm1   Ready     4d
kubevm2   Ready     4d

Once the cluster is up and running let’s see how we can modify the cluster configuration.

Post Deployment Modification of Kubernetes API-server

The entire kubernetes control-plane is deployed as pods. The configuration templates are under /etc/kubernets/manifests directory. This directory is watched by the ‘kubelet’ daemon for changes.

In order to modify the api-server configuration, just edit the kube-apiserver.json file in the manifest directory with required changes and kubelet will restart the api-server with modified options. For example, if you want to experiment with PodSecurityPolicy, edit the /etc/kubernetes/manifests/kube-apiserver.json to enable PodSecurityPolicy.

[snip]
containers": [
      {
        "name": "kube-apiserver",
        "image": "gcr.io/google_containers/kube-apiserver-amd64:v1.4.4",
        "command": [
          "kube-apiserver",
          "--v=2",
          "--insecure-bind-address=127.0.0.1",
          "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,PodSecurityPolicy",
          "--service-cluster-ip-range=10.96.0.0/12",
          "--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--client-ca-file=/etc/kubernetes/pki/ca.pem",
          "--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
          "--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--token-auth-file=/etc/kubernetes/pki/tokens.csv",
          "--runtime-config=extensions/v1beta1/podsecuritypolicy=true",
          "--allow-privileged=true",
          "--secure-port=6443",
          "--advertise-address=192.168.33.10",
          "--etcd-servers=http://127.0.0.1:2379"
        ],
[snip]

Let’s verify if the changes have taken effect or not.

Create a policy to disable privilege containers and then deploy a pod explicitly as a privilege container.

$ kubectl create -f https://raw.githubusercontent.com/bpradipt/examples/master/kubepodsecpolicy/noroot.yaml

$ kubectl create -f https://raw.githubusercontent.com/bpradipt/examples/master/kubeyamls/podex2.yaml
Error from server: error when creating "https://raw.githubusercontent.com/bpradipt/examples/master/kubeyamls/podex2.yaml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: "SETFCAP": capability may not be added]

As you can see how easy it is to experiment with various options. Hope this helps.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...