A Case for Isolating Containers by leveraging Virtualization

All containers running on a specific host, shares the host kernel. While this is fine for a large number of use cases, for certain security focused use cases (eg. Blockchain), this is not acceptable, and a stronger isolation is desired.  This is where isolated containers comes into the picture. In the isolated containers approach, the containers have their own kernel and leverages isolation provided by virtualization mechanism; while retaining the usage, packaging and deployment benefits of a container.

There are multiple works happening in the area of providing stronger isolation to a container by leveraging virtual-machine technology. Intel’s clear containers [1] approach and hyper_ from HyperHQ [2] are few notable approaches.

My team have been dabbling with similar approaches in the lab for quite some time. Our use cases spanned across different CPU architectures (x86_64, ppc64le, s390x). One of the major challenges we encountered with the isolated container approaches was that, these required significant changes to the existing container management solutions or development of newer container management solutions.

This made integration into existing environments costly in terms of effort, skills and investments required.

The integration challenges made us think on alternate mechanisms, which provides both – the necessary isolation guarantees, and easy integration into existing cloud environments. We decided to lay out some fundamental ground rules based on multiple client feedbacks and our experiences, and then worked our way through the entire stack.

Ground rules:

  • Leverage KVM/Qemu for virtualization.
    • KVM/Qemu is available across different CPU architectures and provide us the flexibility to create a solution fine-tuned for specific CPU architectures.
  • Ability to integrate with existing OpenStack environments.
    • Many private cloud deployments are based on OpenStack and integrating into an OpenStack environment was required.
  • 1:1 mapping between container and VM
    • This makes it easier for managing the container life-cycle
  • Ability to use Docker formatted images
    • Docker formatted images are now a de-facto standard for packaging applications. Makes it easier to reuse application images.

Once the ground rules were laid, the next few steps were mostly around figuring out a minimalistic VM image, Qemu optimizations for faster boot, potential Libvirt ,OpenStack and Docker integration points.

The overall workflow is similar to the picture shown below:

isolated-containers-flow

 

For OpenStack integration our approach was leveraging the existing components (like nova-libvirt) as much as possible without the need to write docker specific compute drivers and making the container deployment workflow same as VM.  More details on the possible OpenStack integration points are described in the following OpenStack thread [3]

For cases where OpenStack integration was not needed and Docker integration made more sense, we leveraged the runv approach [4] but instead of N:1 mapping between containers and VM we preferred a 1:1 mapping. This made overall life-cycle management easier, and opened up avenues for future enhancements and optimizations to the isolation mechanism.

There could be other approaches for docker integration. For example, creating a new docker isolation mechanism for qemu/KVM similar to existing mechanism for Hyper-V containers [5] [6] . We are yet to experiment with the docker isolation mechanism.

We also experimented with potential mesos integration for an isolated container workflow and relevant details are available here [7].

I would love to hear your thoughts, questions and experiences. Please feel free to reach out using the comments section below.

[1] https://clearlinux.org/features/clear-containers

[2] http://hypercontainer.io/

[3] http://lists.openstack.org/pipermail/openstack-dev/2016-July/100182.html

[4] https://github.com/hyperhq/runv

[5] https://docs.docker.com/engine/reference/commandline/run/

[6] https://msdn.microsoft.com/en-us/virtualization/windowscontainers/management/hyperv_container

[7] https://docs.google.com/document/d/1_VuFiJqxjlH_CA1BCMknl3sadlTZ69FuDe7qasDIOk0/edit?ts=56fab906

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...