How to use OpenVswitch with Docker

It is a known fact, that Docker uses linux bridge for container networking, by default.

When docker daemon starts, it creates a linux bridge named docker0, and assigns an IP address from the following range of private IP addresses. The containers gets assigned IP addresses from the same subnet.

“172.17.42.1/16”,  “10.0.42.1/16”, “10.1.42.1/16”, “10.42.42.1/16”, “172.16.42.1/24”, “172.16.43.1/24”, “172.16.44.1/24”, “10.0.42.1/24”, “10.0.43.1/24”, “192.168.42.1/24”, “192.168.43.1/24”, “192.168.44.1/24”.

The IP address, that gets assigned depends on IP addresses already in use by the docker host. For example, if none of the private IP addresses in the list above is in use by the docker host, then the docker daemon will assign docker0 the IP address 172.17.42.1, and the containers from same subnet (172.17.42.1/16). Note that IP allocation to the containers is handled by the docker daemon itself.

It’s also possible to use a different Linux bridge instead of the default docker0. The docker daemon needs to be started mentioning the same. All containers will then use that bridge for connectivity.

For normal usage Linux bridge is a good choice. However, there are cases, where OpenvSwitch (OVS) might be required instead of Linux bridge. For example, a single Linux bridge can only handle 1024 ports. This limits the scalability of docker as it won’t be possible to create more than 1024 containers, each having a single network interface. Another example is requirement of tunneling mechanism like GRE or VXLAN. But as of this writing, is not natively integrated with docker. There is an issue opened for the same- https://github.com/docker/docker/issues/8952

However, it is possible to use OVS for docker networking. Let’s see how.

First ,we need to understand what happens internally when a docker container is started.

This is what happens behind the scenes:

  1. Creation of veth pair
  2. Attaching one end of veth to the host bridge (docker0), and putting another end in container net namespace.
  3. Rename the veth end-point in container net namespace as ethX.
  4. IP address configuration for the containers

For Linux bridge, the steps are taken care by docker daemon itself, hence there is no user intervention required.

Logically, the relationship looks like the following:

 

docker-bridge

Since OVS is not yet integrated natively with docker, we have to perform the above steps manually.

Let’s go through the steps for a specific scenario of using OVS bridge ovsbr0 for docker containers. The instructions mentioned here are not architecture specific and works for both Intel and Power archs. My setup is a mix of Ubuntu 14.10 and 15.04 on Intel and Power KVM.

1. Create OVS bridge

[host]#ovs-vsctl add-br ovsbr0

2. Configure IP address

[host]#ip addr add dev ovsbr0 10.10.0.1/16
[host]#ip link set dev ovsbr0 up

Depending on your requirement, you might want to use DHCP. Accordingly, you might be required to setup a DHCP server. For example this is my DHCP server configuration on Ubuntu 15.04 LE (ppc64el) server.

[host]#cat /etc/default/isc-dhcp-server
INTERFACES="ovsbr0"
[host]#cat /etc/dhcp/dhcpd.conf
subnet 10.10.0.0 netmask 255.255.0.0 {
  range 10.10.0.2 10.10.0.254;
  option routers 10.10.0.1;
}

3. Create veth pair (veth_hostc0 – veth_contc0)

[host]# ip link add veth_hostc0 type veth peer name veth_contc0

4. Attaching one end of the veth pair (veth_hostc0) to host bridge (ovsbr0) and putting another end (veth_contc0) to the container net namespace.

Attach veth_hostc0 to ovsbr0

[host]# ovs-vsctl add-port ovsbr0 veth_hostc0

Create a container without default networking

[host]#docker run --net='none' -itd busybox /bin/sh
6a37777816a7073d1d238161e1f88d4a8e5032f26209b1dfacdaad69dea99606

Put veth_contc0 in the container net name space.

[host]#docker inspect -f '{{.State.Pid}}' 6a37777816a7
3975
[host]#mkdir -p /var/run/netns
[host]#ln -s /proc/3975/ns/net /var/run/netns/3975
[host]#ip link set veth_contc0 netns 3975

Check if the veth end point is available in the container net namespace.

[host]# docker attach 6a37777816a7
[container]# ifconfig -a
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth_contc0 Link encap:Ethernet  HWaddr 02:84:C7:43:7D:62  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:47 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4450 (4.3 KiB)  TX bytes:4450 (4.3 KiB)

5. Rename the veth end-point in container net namespace as ethX.

[host]#ip netns exec 3975 ip link set dev veth_contc0 name eth0
[host]#ip netns exec 3975 ip link set dev eth0 up

Check the device name inside the container

[host]# docker attach 6a37777816a7
[container]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 02:84:C7:43:7D:62  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:47 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4450 (4.3 KiB)  TX bytes:4450 (4.3 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

6. IP address configuration for the containers

If static IP is used, then you would be required to set an IP for the container ethX interface explicitly. Otherwise, the container image needs to have a DHCP client (like dhclient, udhcpc) to get IP from the DHCP server.

Hope this gives you an idea on how to use OVS with docker.

Beyond few containers, you will soon realize, that the manual steps are too tedious to follow practically. You’ll need some automation.

Fortunately there is a way out, courtesy by @jpetazzo, who has written an excellent script to automate the entire process.

Download the script from https://github.com/jpetazzo/pipework.git

[host]#git clone https://github.com/jpetazzo/pipework.git

The manual set of steps mentioned above boils down to the following single command

[host]#pipework ovsbr0 -i eth0 $(docker run --net='none' busybox /bin/sh) dhcp

I have been using this to create 1000+ docker containers without any issues.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...