Tracing and Profiling Microservices Application Deployed on IBM Cloud Private

In this article, we’ll see how to trace and view platform instrumentation (profiling) data with OpenTracing and Zipkin for a sample micro-services based application deployed on IBM Cloud Private (ICP).

Examples of platform specific instrumentation data are processor execution efficiency (cycles per instruction), memory bus bandwidth, I/O bus bandwidth and so on. There are scenarios where a combination of application and platform specific instrumentation data is of immense help to definitively identify issues. For example, in the case of memory-intensive workload, there could be a scenario where memory capacity is available, however, memory bus bandwidth might be starved, thereby negatively affecting the performance. Having platform instrumentation data available in the context of application is very helpful in identifying the root cause of the performance issue.

Overview of technologies involved

OpenTracing is an open and vendor-neutral distributed tracing standard for applications. It allows developers to instrument their code in a vendor neutral way. Zipkin is a distributed tracing system and supports OpenTracing. For detailed information, please refer OpenTracing documentation.

The following diagram gives an overview of the components involved when using OpenTracing with Zipkin backend. The diagram also shows the components involved in extracting platform instrumentation data from hardware performance monitoring counters (PMCs).

opentracing_zipkin_arch

If you are interested in knowing more about our work related to leveraging platform specific instrumentation data in OpenTracing please refer to the following article – https://goo.gl/Jdm5an

IBM Cloud Private (ICP), which is built on open source technologies, including Kubernetes, provides a platform that you can leverage to develop modern applications. You can read more about IBM Cloud Private here.

Pre-requisites
Following are the pre-requisites:

  1. An IBM Cloud Private 2.1 installation with 1 master and at least 1 worker node. See Installing.
  2. Install the Kubernetes command line kubectl and configure it to connect to your IBM Cloud Private instance. See Accessing your IBM Cloud Private cluster by using the kubectl CLI.
  3. Configure the private Docker registry. See Pushing and pulling images.

Sample application
Sock-shop is a very popular microservices demo application simulating an e-commerce website selling socks. The following diagram gives an overview of the application.
sockshop-arch

Src- https://github.com/microservices-demo/microservices-demo.github.io/blob/master/assets/Architecture.png

Adding platform specific instrumentation data
In order to enable capturing of platform instrumentation data in your application with OpenTracing, following snippet needs to be added to your application code. Currently only ‘golang’ applications are supported with work underway to add support for Java.

In your application code, initialize OpenTracing backend with perfevents observer

obs := perfevents.NewObserver()
tracer := zipkin.NewTracer(… , zipkin.WithObserver(obs), )

Specify the platform instrumentation data you need to collect as part of the trace span.

tracer.StartSpan(“new”, opentracing.Tag{“perfevents”, “cpu-cycles, instructions”})

This is all that is required to start collecting platform instrumentation data as part of your traces.

For the purpose of this article, the ‘catalogue’ service was modified to add platform instrumentation ( cpu-cycles and instructions) data in the traces. The modified code is available from the following github link – https://github.com/bpradipt/catalogue

Deploying the sample micro-services application on IBM Cloud Private
You can use the following Kubernetes yaml that comes with the application to deploy it on any Kubernetes cluster, including IBM Cloud Private.
https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml

However, for the purpose of this article, we have used the following steps to deploy the sock-shop application in a heterogeneous cluster consisting of Intel and Power (ppc64le) nodes and platform instrumentation enabled for the ‘catalogue’ service.

Steps

  1. Populate the IBM Cloud Private Docker registry with the required Docker images$ wget https://raw.githubusercontent.com/bpradipt/microservices-demo/ppc64le/deploy/kubernetes/image_bom.txt
    image_bom.txt contains the list of required Docker images$ wget https://raw.githubusercontent.com/bpradipt/microservices-demo/ppc64le/deploy/kubernetes/push_to_icp
    $ chmod +x push_to_icp
    $ ./push_to_icp [icp_private_registry]In my setup, the Docker registry was available at mycluster.icp
    $ ./push_to_icp mycluster.icp
    This will ensure all the required Docker images are available in the IBM Cloud Private Docker registry.
  2. Build the modified ‘catalogue’ service code$ git clone https://github.com/bpradipt/catalogue.git
    $ cd catalogue
    $ make image-ppc64le
    This will create the catalogue Docker image (weaveworksdemos/catalogue) for Power (ppc64le).Push the image to the IBM Cloud Private Docker registry.
    $ docker tag weaveworksdemos/catalogue mycluster.icp:8500/sockshop/catalogue-ppc64le
    $ docker push mycluster.icp:8500/sockshop/catalogue-ppc64le
  3. Download the multi-arch deployment yaml$ wget https://raw.githubusercontent.com/bpradipt/microservices-demo/ppc64le/deploy/kubernetes/complete-demo-multi-arch.yaml
    $ kubectl create –f complete-demo-multi-arch.yamlThe YAML assumes the images are available under ‘sockshop’ namespace in the IBM Cloud Private Docker registry – mycluster.icp:8500

On successful deployment, you can retrieve the details of the front-end service to access the application and the zipkin service for accessing the traces.

This is a short video showing the deployment and tracing in action.

Hope you’ll find this useful. Thanks to Hemant K Shaw for helping with the demo setup. He is also one of the key developers working on adding platform instrumentation support to OpenTracing.

[This article was first published at – https://developer.ibm.com/linuxonpower/2017/12/22/profile-application-ibm-cloud-private/]

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...