Bootstrapping an ETCD cluster for Kubernetes

In this article we’ll look at the step-by-step instructions to bootstrap an etcd cluster for Kubernetes. We’ll use a 3-member etcd cluster.

Take a look at the following article for etcd design considerations when planning for a Kubernetes deployment – https://goo.gl/n5VX1f

ETCD supports different mechanism for bootstrapping a cluster. However, in this article we’ll specifically look at the ‘static’ bootstrapping method which is sufficient for a Kubernetes deployment. This is a working example from a RHEL 7 installation.

For the purpose of illustration here are the FQDN and IPs for the 3-members of the etcd cluster:

pkb-rhel71-1.kube.com (192.168.122.124)
pkb-rhel71-2.kube.com (192.168.122.125)
pkb-rhel71-3.kube.com (192.168.122.126)

Certificate Generation  (Optional)

These steps are only required when using self-signed certificates. Perform these steps on any Linux system with openssl installed.

Create Certificate Authority (CA)

# mkdir certs
# cd certs
# openssl genrsa -out ca-key.pem 2048
# openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=my-ca"
[Replace CN as appropriate]

Generate ETCD Member Key-pair
In this section we will generate a unique certificate and key for every member of the cluster.
Create etcd-openssl.cnf file as follows:

[req] 
req_extensions = v3_req 
distinguished_name = req_distinguished_name 
[req_distinguished_name] 
[ v3_req ] 
basicConstraints = CA:FALSE 
keyUsage = nonRepudiation, digitalSignature, keyEncipherment 
subjectAltName = @alt_names[alt_names] 
IP.1 = $ENV::MEMBER_IP 

Generate the Certificates

# export MEMBER_FQDN=pkb-rhel71-1.kube.com
# export MEMBER_IP=192.168.122.124
# openssl genrsa -out ${MEMBER _FQDN}-key.pem 2048 
# openssl req -new -key ${MEMBER_FQDN}-key.pem -out ${MEMBER_FQDN}.csr -subj "/CN=${MEMBER_FQDN}" -config etcd-openssl.cnf 
# openssl x509 -req -in ${MEMBER_FQDN}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${MEMBER_FQDN}.pem -days 365 -extensions v3_req -extfile etcd-openssl.cnf 

Generate certificates for all the 3 members by changing the MEMBER_FQDN and MEMBER_IP accordingly.

Install ETCD on all the 3 Members

Etcd is usually shipped with most of the Linux distros, consequently installing etcd is just about installing the right package for your distribution.

If you are using RHEL/CentOS 7 Little Endian (LE), then you can download the etcd packages from the Unicamp repo – http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/misc_ppc64el/

If using self-signed certificates (as described previously), then copy the respective certificates for each member to a well-defined path. In the example described herewith, the certificates are in /etc/ssl/etcd directory. Ensure that the directory and the contents are owned by etcd:etcd

Separate Partition for Data Directory

Use a separate partition (or disk) for the data dir. In this example, for each of the member host, a separate disk  is mounted at /etcd_data

Ensure /etcd_data is owned by ‘etcd:etcd’ .

ETCD Configuration File

A complete working configuration file for one of the member of the example cluster is described below. The config file on RHEL can be found at /etc/etcd/etcd.conf.

The texts highlighted in bold needs to be changed for every member.

# [member]
ETCD_NAME=pkb-rhel71-1
ETCD_DATA_DIR=/etcd_data
#ETCD_WAL_DIR=""
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""

#[cluster]
ETCD_INITIAL_CLUSTER="pkb-rhel71-1=https://pkb-rhel71-1.kube.com:2380,pkb-rhel71-3=https://pkb-rhel71-3.kube.com:2380,pkb-rhel71-2=https://pkb-rhel71-2.kube.com:2380"
ETCD_INITIAL_CLUSTER_STATE=new

ETCD_INITIAL_ADVERTISE_PEER_URLS=https://pkb-rhel71-1.kube.com:2380
ETCD_ADVERTISE_CLIENT_URLS=https://pkb-rhel71-1.kube.com:2379

ETCD_LISTEN_PEER_URLS=https://pkb-rhel71-1.kube.com:2380
ETCD_LISTEN_CLIENT_URLS="https://pkb-rhel71-1.kube.com:2379"

#[proxy]
ETCD_PROXY="off"

#[security]
ETCD_CERT_FILE=/etc/ssl/etcd/pkb-rhel71-1.kube.com.pem
ETCD_KEY_FILE=/etc/ssl/etcd/pkb-rhel71-1.kube.com-key.pem
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/pkb-rhel71-1.kube.com.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/pkb-rhel71-1.kube.com-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem

Start ETCD Service

# service etcd restart

Verify ETCD

#etcdctl -C https://pkb-rhel71-1.kube.com:2379 --cert-file /etc/ssl/etcd/pkb-rhel71-1.kube.com.pem --key-file /etc/ssl/etcd/pkb-rhel71-1.kube.com-key.pem --ca-file /etc/ssl/etcd/ca.pem  cluster-health

member 26c7cf115990a094 is healthy: got healthy result from https://pkb-rhel71-2.kube.com:2379
member 41b807391e1278ca is healthy: got healthy result from https://pkb-rhel71-3.kube.com:2379
member 5d9f224fdf4b5d89 is healthy: got healthy result from https://pkb-rhel71-1.kube.com:2379

Hope you find this article useful when setting up an etcd cluster.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...