In our Kubernetes experiments, sometimes we have the need to profile the deployed applications using Linux ‘perf’. We were looking for a mechanism where the ‘perf’ data capture was specific to the deployed application and fully automatic, without any manual intervention.
This is when we thought of leveraging the automatic sidecar injector pattern which is heavily used in Istio (service mesh). Few of the team members, namely Hemant and Christy, being part of Istio development community really helped to dig deeper into the nuances.
If you have a similar use case, this article might be helpful.
Automatic sidecar injection is actually a mutating admission webhook which are invoked during the mutation phase and allows modification of the resource object.
The ‘perf’ sidecar injector is a mutating admission webhook which either uses an annotation or namespace label to add a sidecar to the pod.
The following picture shows a high level overview of how the POD gets modified (mutated) as part of the admission process.
Additionally the sidecar injector leverages PID sharing to automatically capture the ‘perf’ data for the POD processes.
Ensure you are using Kubernetes 1.10+ and the following settings are enabled:
- Feature-gate PodShareProcessNamespace=true turned on for both apiserver and kubelet
- Ensure kube-apiserver has the admission-control flag set with MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers added
$ kubectl api-versions | grep admissionregistration admissionregistration.k8s.io/v1beta1
For IBM Cloud Private, you can use the kube_apiserver_extra_args and kubelet_extra_args configuration parameters during install.
More details on IBM Cloud Private install time config options is available from the following link .
Before getting into the setup details, let’s take a quick look at the ‘perf’ container which gets added as a sidecar via the injector.
The source of the perf container is available from the following github link — https://github.com/bpradipt/perf-container
The perf container can be run with the following options:
entrypoint.sh <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]>
- stat: runs perf stat command
- record: runs perf record command
- probe: runs perf probe command
- trigger : The ‘perf’ command can be started in response to a trigger — create/open/modify events for file /tmp/startperf
- max-run-time: is the max time the perf command will be run. Default is 60 sec
A sample invocation looks like this:
perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g
This captures perf stat for PIDs — 1, 3, 8 including call-graph and dumps to /out/perf-stat-output-abcxyz
Deploy the ‘Perf’ Sidecar Injector
- Clone the source
$ git clone https://github.com/bpradipt/perf-sidecar-injector.git $ cd perf-sidecar-injector
- Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment
$ ./deployment/webhook-create-signed-cert.sh \ --service perf-sidecar-injector-webhook-svc \ --secret perf-sidecar-injector-webhook-certs \ --namespace default
- Patch the MutatingWebhookConfiguration by setting caBundle with correct value from Kubernetes cluster
$ cat deployment/mutatingwebhook.yaml | \ deployment/webhook-patch-ca-bundle.sh > \ deployment/mutatingwebhook-ca-bundle.yaml
- Deploy sidecar injector resources
Edit the configmap as per your requirement. By default the ‘perf’ data is kept in the hostPath — ‘/out’
$ kubectl create -f deployment/configmap-record.yaml $ kubectl create -f deployment/deployment.yaml $ kubectl create -f deployment/service.yaml $ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
Check if the sidecar injector webhook is running by
$ kubectl get pods NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 5m
There are two ways by which the webhook can be instructed to add the ‘perf’ sidecar:
- annotation: Adding ‘perf-sidecar-injector-webhook/inject: “yes”‘ annotation to the POD spec will ensure automatic injection of ‘perf’ sidecar.
- namespace label: Adding ‘sidecar-injector=enabled’ label to a specific namespace will ensure automatic injection of ‘perf’ sidecar to every POD deployed in this namespace
Let’s use the namespace label option such that any POD deployed to the ‘default’ namespace will have the ‘perf’ sidecar added automatically.
- Add label
$ kubectl label namespace default sidecar-injector=enabled $ kubectl get namespace -L sidecar-injector NAME STATUS AGE SIDECAR-INJECTOR default Active 18h enabled kube-public Active 18h kube-system Active 18h
- Deploy a sample application and collect ‘perf’ data
You can use ‘perf report’ or other visualization tools to view the captured ‘perf’ data
Take a look at the following video for a complete demonstration:
[This article was first published here – https://medium.com/ibm-cloud/profiling-applications-deployed-on-kubernetes-using-perf-and-sidecar-injector-a3eeecb0c3e3]