Howto Configure flannel Overlay Network with VXLAN for Docker on Power Servers

Flannel provides a configurable virtual overlay network for use with containers. For a multi-host container cluster this provides an easy way to have a routable IP address inside the cluster. Flannel uses etcd distributed key-value store to save the network configurations and make it available to different hosts in the cluster. In this article we’ll see how to setup flannel with VXLAN backend and use it with docker on Power servers. More technical details on flannel is available on it’s github page.

Getting etcd  and flannel
etcd can be built from source or binary packages for Ubuntu 16.04 LE and Fedora 24 LE are available from distribution repositories. Binary packages for RHEL 7 LE is available from the Unicamp repo.

Similarly flannel can also be built from source. Binary packages for Ubuntu and Fedora are not yet available. However binary packages for RHEL 7 LE is available from the Unicamp repo.

Setup etcd
If installing via rpm or deb packages, then etcd configuration can be found under /etc/etcd/etcd.conf for RedHat based systems and under /etc/default/etcd.conf for Ubuntu
Ensure you atleast setup ETCD_LISTEN_CLIENT_URLS and ETCD_ADVERTISE_CLIENT_URLS correctly so that the remote clients can reach the etcd server.
My test setup have the following values:

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:4001"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:4001, http://192.168.122.173:4001"

Start etcd
Start the etcd daemon using the following command:

# systemctl restart etcd

Setup flannel Network Configuration in etcd
Create a network configuration JSON file for the overlay network. An example configuration is shown below:

# cat flannel-config.json 
{
  "Network": "10.20.0.0/16",
  "SubnetLen": 24,
  "Backend": {
    "Type": "vxlan",
    "VNI": 1
  }
}

The ‘Type’ attribute specifies the VXLAN backend. This requires the host kernel to have VXLAN support. A quick way to validate VXLAN support in the host kernel is by running the following command:

# cat /boot/config-`uname -r` | grep CONFIG_VXLAN
CONFIG_VXLAN=m

Add the network configuration to etcd

# etcdctl set kubernetes-cluster/network/config < flannel-config.json

Feel free to use an etcd key prefix of your choice instead of kubernetes-cluster/network.
The etcd entry can be validated by running the following command:

# etcdctl get kubernetes-cluster/network/config 
{
  "Network": "10.20.0.0/16",
  "SubnetLen": 24,
  "Backend": {
    "Type": "vxlan",
    "VNI": 1
  }
}

Start flannel
At a minimum, the location of etcd server as well as the etcd key prefix needs to be provided to the flannel daemon. If running flannel via systemd then the values needs to be provided via the appropriate configuration file. For RedHat based systems the configuration file is /etc/sysconfig/flanneld. Ensure the values for FLANNEL_ETCD and FLANNEL_ETCD_KEY is correctly specified.
Here are the values from my test setup:

FLANNEL_ETCD="http://192.168.122.173:4001"
FLANNEL_ETCD_KEY="/kubernetes-cluster/network"

Start the flannel daemon using the following command:

# systemctl restart flanneld

Following is an example of running flannel daemon directly:

# flanneld -etcd-endpoints=http://192.168.122.173:4001 -etcd-prefix=kubernetes-cluster/network

If you are planning to setup flannel in Ubuntu, then you might want to use the following contents and create flanneld.service under /lib/systemd/system/flanneld.service

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=-/etc/default/flanneld
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

The /etc/default/flanneld configuration file looks like the following:

# Flanneld configuration options
# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://192.168.122.173:4001"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/kubernetes-cluster/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

Configure docker to use flannel
Flannel daemon writes host specific flannel network configuration under /run/flannel/subnet.env. For example here is the output from two systems in my test setup:

# hostname
pkb-rhel71-1.kube.com
# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.20.0.0/16
FLANNEL_SUBNET=10.20.31.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
# hostname
pkb-ubuntu1604-1.kube.com
# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.20.0.0/16
FLANNEL_SUBNET=10.20.48.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

On each host, values for FLANNEL_SUBNET and FLANNEL_MTU needs to be used with --bip and --mtu docker daemon options respectively. For example here are the docker daemon options for my RHEL server.

# cat /etc/sysconfig/docker
other_args="--bip=10.20.48.1/24 --mtu=1450"

Similary, here are the docker daemon options for my Ubuntu server.

# cat /etc/default/docker 
DOCKER_OPTS="--bip=10.20.31.1/24 --mtu=1450"

This is all that is required to setup flannel with vxlan for use with docker. Hope this helps.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...