Howto Enforce Cluster wide Policies for a Kubernetes based Docker Cluster

For enterprise production deployments of docker clusters, enforcing cluster wide policies to restrict what a container is allowed to do is an extremely important requirement. For kubernetes based clusters this is where PodSecurityPolicy comes into the picture. A PodSecurityPolicy object controls the actions that a kubernetes POD can perform. To put it simply if you want to specify what a docker container is allowed to do in your kubernetes cluster, PodSecurityPolicy is what you need to look at.

From the kubernetes documentation –

The PodSecurityPolicy allows an administrator to control the following:

• Running of privileged containers.
• Capabilities a container can request to be added.
• The SELinux context of the container.
• The user ID.
• The use of host namespaces and networking.
• Allocating an FSGroup that owns the pod’s volumes
• Configuring allowable supplemental groups
• Requiring the use of a read only root file system
• Controlling the usage of volume types

More details are available here – http://kubernetes.io/docs/user-guide/pod-security-policy/

In this article we’ll look at configuration examples to set cluster wide POD policies for a kubernetes cluster. These instructions applies to kubernetes running on any platform – Intel, PowerPC, ARM etc..

API Server Configuration

For PODSecurityPolicy to work the following configs needs to be enabled for the API server

Enable API extensions
The following parameter needs to be added to the apiserver startup arguments “--runtime-config=extensions/v1beta1/podsecuritypolicy=true”

On RedHat based systems this parameter needs to be set for the KUBE_API_ARGS variable in /etc/kubernetes/apiserver file

Enable PodSecurityPolicy admission control policy

The following parameter needs to be added to the apiserver startup arguments  “--admission-control=PodSecurityPolicy”

On RedHat based systems this parameter needs to be set for the KUBE_ADMISSION_CONTROL variable in /etc/kubernetes/apiserver file

Applying Policies

The policies are defined in YAML or JSON files and applied using kubectl command

# kubectl create –f  [policy_file]

The example YAMLs used in this article can be found here – https://github.com/bpradipt/examples

Example Policies

1. Don’t allow process(es) inside the container to run as the ‘root’ user

{
  "kind": "PodSecurityPolicy",
  "apiVersion":"extensions/v1beta1",
  "metadata": {
    "name": "noroot"
  },
  "spec": {
      "privileged": false,
      "seLinux": {
          "rule": "RunAsAny"
      },
      "supplementalGroups": {
          "rule": "RunAsAny"
      },
      "runAsUser": {
          "rule": "MustRunAsNonRoot"
      },
      "fsGroup": {
          "rule": "RunAsAny"
      },
      "volumes": ["*"]
  }
}

For a POD to be successfully deployed with the above policy in force, following conditions should be met:

• The POD container image(s) should have USER attribute defined
OR
• The POD YAML file should explicitly specify the non-root USER ID as part of securityContext

Here is an example of POD YAML file:

apiVersion: "v1"
kind: "Pod"
metadata:
 name: "mysql"
 labels:
   name: "db"
spec:
    containers:
      - name: "mysql"
        image: "ppc64le/mysql"
        imagePullPolicy: "IfNotPresent"
        securityContext:
           runAsUser: 102
        env:
           - name: MYSQL_ROOT_PASSWORD
             value: password
        ports:
          - containerPort: 3306

Note that USER ID needs to be used and not USER Name.

When the above policy is in force, the following behaviour is observed for various scenarios.

Scenario-1
Container image doesn’t have USER instruction and POD YAML doesn’t specify a non-root user id.

# kubectl get pods
NAME  READY STATUS             RESTARTS   AGE
mysql 0/1   VerifyNonRootError 0          7s

Following is the error text:

Error syncing pod, skipping: failed to “StartContainer” for “mysql” with VerifyNonRootError: “container has no runAsUser and image will run as root”

Scenario-2
Container image having USER instruction.

Following error is noticed.

failed to “StartContainer” for “mysql” with VerifyNonRootError: “can’t tell if image runs as root: non-numeric user (mysql) is not allowed”

Scenario-3
When using runAsUser: 0 (root user id) in POD YAML, error will be thrown when trying to create a POD

# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden by the pod security policy mysql]

Scenario-4
When using privileged: true in POD YAML, error will be thrown when trying to create a POD

# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

2. Don’t allow process(es) inside the container to run as the ‘root’ user and drop specific capabilities

{
  "kind": "PodSecurityPolicy",
  "apiVersion":"extensions/v1beta1",
  "metadata": {
    "name": "dropcapabilities"
  },
  "spec": {
      "privileged": false,
      "requiredDropCapabilities": ["SETFCAP", "DAC_OVERRIDE"],
      "seLinux": {
          "rule": "RunAsAny"
      },
      "supplementalGroups": {
          "rule": "RunAsAny"
      },
      "runAsUser": {
          "rule": "MustRunAsNonRoot"
      },
      "fsGroup": {
          "rule": "RunAsAny"
      },
      "volumes": ["*"]
  }
}

With the above policy in place, if a user tries to deploy a POD with the following YAML an error will be thrown.

apiVersion: "v1"
kind: "Pod"
metadata:
 name: "mysql"
 labels:
   name: "db"
spec:
    containers:
      - name: "mysql"
        image: "ppc64le/mysql"
        imagePullPolicy: "IfNotPresent"
        securityContext:
           privileged: true
           capabilities:
              add: ["SETFCAP"]
        env:
           - name: MYSQL_ROOT_PASSWORD
             value: password
ports:
          - containerPort: 3306
# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: "SETFCAP": capability may not be added]

As you can see from the above examples, PodSecurityPolicy is a very powerful functionality especially suited for enterprise deployments with strict compliance requirements.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...