Profiling Applications Deployed on Kubernetes using ‘Perf’ and Sidecar Injector

In our Kubernetes experiments, sometimes we have the need to profile the deployed applications using Linux ‘perf’. We were looking for a mechanism where the ‘perf’ data capture was specific to the deployed application and fully automatic, without any manual intervention.
This is when we thought of leveraging the automatic sidecar injector pattern which is heavily used in Istio (service mesh). Few of the team members, namely Hemant and Christy, being part of Istio development community really helped to dig deeper into the nuances.
If you have a similar use case, this article might be helpful.
Automatic sidecar injection is actually a mutating admission webhook which are invoked during the mutation phase and allows modification of the resource object.
The ‘perf’ sidecar injector is a mutating admission webhook which either uses an annotation or namespace label to add a sidecar to the pod.
The following picture shows a high level overview of how the POD gets modified (mutated) as part of the admission process.



Additionally the sidecar injector leverages PID sharing to automatically capture the ‘perf’ data for the POD processes.

Ensure you are using Kubernetes 1.10+ and the following settings are enabled:

    • Feature-gate PodShareProcessNamespace=true turned on for both apiserver and kubelet
    • Ensure kube-apiserver has the admission-control flag set with MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers added
$ kubectl api-versions | grep admissionregistration

For IBM Cloud Private, you can use the kube_apiserver_extra_args and kubelet_extra_args configuration parameters during install.
More details on IBM Cloud Private install time config options is available from the following link .
Before getting into the setup details, let’s take a quick look at the ‘perf’ container which gets added as a sidecar via the injector.
The source of the perf container is available from the following github link —
The perf container can be run with the following options: <stat|record > <trigger[1|0]> <max-run-time[X]> <repeat-count[N]> 


  • stat: runs perf stat command
  • record: runs perf record command
  • probe: runs perf probe command
  • trigger : The ‘perf’ command can be started in response to a trigger — create/open/modify events for file /tmp/startperf
  • max-run-time: is the max time the perf command will be run. Default is 60 sec

A sample invocation looks like this:

perf stat -a -p 1,3,8 -x ',' -I 1000 -A -o /out/perf-stat-output-abcxyz --append -g

This captures perf stat for PIDs — 1, 3, 8 including call-graph and dumps to /out/perf-stat-output-abcxyz

Deploy the ‘Perf’ Sidecar Injector

  • Clone the source
$ git clone
$ cd perf-sidecar-injector
  • Create a signed certificate/key pair and store it in a Kubernetes secret that will be consumed by the sidecar deployment
$ ./deployment/ \
    --service perf-sidecar-injector-webhook-svc \
    --secret perf-sidecar-injector-webhook-certs \
    --namespace default
  • Patch the MutatingWebhookConfiguration by setting caBundle with correct value from Kubernetes cluster
$ cat deployment/mutatingwebhook.yaml | \
    deployment/ > \
  • Deploy sidecar injector resources
    Edit the configmap as per your requirement. By default the ‘perf’ data is kept in the hostPath — ‘/out’
$ kubectl create -f deployment/configmap-record.yaml
$ kubectl create -f deployment/deployment.yaml
$ kubectl create -f deployment/service.yaml
$ kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
  • Verify
    Check if the sidecar injector webhook is running by
$ kubectl get pods
NAME                                                  READY     STATUS    RESTARTS   AGE
sidecar-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m
$ kubectl get deployments
NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
sidecar-injector-webhook-deployment   1         1         1            1           5m

There are two ways by which the webhook can be instructed to add the ‘perf’ sidecar:

  1. annotation: Adding ‘perf-sidecar-injector-webhook/inject: “yes”‘ annotation to the POD spec will ensure automatic injection of ‘perf’ sidecar.
  2. namespace label: Adding ‘sidecar-injector=enabled’ label to a specific namespace will ensure automatic injection of ‘perf’ sidecar to every POD deployed in this namespace

Let’s use the namespace label option such that any POD deployed to the ‘default’ namespace will have the ‘perf’ sidecar added automatically.

  • Add label
$ kubectl label namespace default sidecar-injector=enabled
$ kubectl get namespace -L sidecar-injector
default       Active    18h       enabled
kube-public   Active    18h
kube-system   Active    18h
  • Deploy a sample application and collect ‘perf’ data

You can use ‘perf report’ or other visualization tools to view the captured ‘perf’ data
Take a look at the following video for a complete demonstration:

[This article was first published here –]

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...