Kubernetes ETCD Data Store Cluster with TLS Certificate. 4


Kubernetes Secure ETCD Data Store Cluster with TLS Certificate configuration Step by Step.

This article is all about configuration of ETCD kubernetes data store with high security  measurement. We will use TLS/SSL certificate so that all communication within cluster will secure and private.

Environment:

3 RHEL 7 / Centos 7 Server with minimal installation.
etcd and etcdctl binary which we can download from https://github.com/coreos/etcd/releases/
openssl.conf will provide in tutorial.

Inventory:

Node1: etcd-01.syshunt.com 192.168.43.45
Node2: etcd-02.syshunt.com 192.168.43.46
Node3: etcd-03.syshunt.com 192.168.43.47

Certificate Generation:

We will use openssl tool for certificate generation as it required for secure communication.

Create Certificate Authority (CA):

Create “etcd-certificate” directory and switch in etcd-certificate directory, will use this directory in our tutorial for all certificate generation,

[[email protected]]# mkdir /root/etcd-certificate
[[email protected]]# cd /root/etcd-certificate
[[email protected] etcd-certificate]# openssl genrsa -out ca-key.pem 2048
[[email protected] etcd-certificate]# openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=etcd-ca"

ETCD node-1 certificate generation:

Create openssl.conf file with below given configuration,

[[email protected] etcd-certificate]# vi openssl.conf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-01
IP.1 = 192.168.43.45
IP.2 = 127.0.0.1

Set openssl.conf location variable in your current shell,

[[email protected] etcd-certificate]# CONFIG=`echo $PWD/openssl.conf`

Generate Certificates:

[[email protected] etcd-certificate]# openssl genrsa -out member-etcd-01-key.pem 2048
[[email protected] etcd-certificate]# openssl req -new -key member-etcd-01-key.pem -out member-etcd-01.csr -subj "/CN=etcd-01" -config ${CONFIG}
[[email protected] etcd-certificate]# openssl x509 -req -in member-etcd-01.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-01.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}

ETCD node-2 certificate generation:

Create openssl.conf file with below given configuration,

[[email protected] etcd-certificate]# vi openssl.conf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-02
IP.1 = 192.168.43.46
IP.2 = 127.0.0.1

Generate Certificates:

[[email protected] etcd-certificate]# openssl genrsa -out member-etcd-02-key.pem 2048
[[email protected] etcd-certificate]# openssl req -new -key member-etcd-02-key.pem -out member-etcd-02.csr -subj "/CN=etcd-02" -config ${CONFIG}
[[email protected] etcd-certificate]# openssl x509 -req -in member-etcd-02.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-02.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}

ETCD node-3 certificate generation:

Create openssl.conf file with below given configuration,

[[email protected] etcd-certificate]# vi openssl.conf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-03
IP.1 = 192.168.43.47
IP.2 = 127.0.0.1

Generate Certificates:

[[email protected] etcd-certificate]# openssl genrsa -out member-etcd-03-key.pem 2048
[[email protected] etcd-certificate]# openssl req -new -key member-etcd-03-key.pem -out member-etcd-03.csr -subj "/CN=etcd-03" -config ${CONFIG}
[[email protected] etcd-certificate]# openssl x509 -req -in member-etcd-03.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-03.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}

Copy all created certificates in every etcd nodes from “/etc/ssl/etcd-certificate” directory to “/etc/ssl/etcd/ssl/”:

[[email protected]* etcd]# cp -rvp /root/etcd-certificate/*.pem /etc/ssl/etcd/ssl/

Install ETCD binary on all the 3 etcd Members:

Copy etcd and etcdctl binary on all 3 etcd members which you have downloaded form
etcd github project page ( https://github.com/coreos/etcd/releases/ ),

[[email protected]* etcd]# cp -vp etcd etcdctl /usr/bin

Create Etcd data directory (on all 3 members):

Etcd data directory is very important as etcd utility saves all cluster data in data directory,

[[email protected]* etcd]# mkdir /var/lib/etcd

Create etcd user on all 3 nodes:

[[email protected]* etcd]# useradd etcd -s /sbin/nologin -r -d /var/lib/etcd

Fixing permissions on all etcd nodes at required directory and files:

[[email protected]* etcd]# chmod -Rv 550 /etc/ssl/etcd/
[[email protected]* etcd]# chmod 440 /etc/ssl/etcd/ssl/*.pem
[[email protected]* etcd]# chown -Rv etcd:etcd /etc/ssl/etcd/
[[email protected]* etcd]# chown -Rv etcd:etcd /etc/ssl/etcd/*
[[email protected]* etcd]# chown etcd:etcd /var/lib/etcd/

Setting Up etcd service daemon files on all etcd nodes:

It is required to create etcd service files on every etcd nodes so that we can start and enable etcd as service daemon.

[[email protected]* etcd]# vi /usr/lib/systemd/system/docker.service

[Unit]
Description=etcd
After=network.target

[Service]
Type=notify
User=etcd
EnvironmentFile=/etc/etcd.env
ExecStart=/usr/bin/etcd
NotifyAccess=all
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

ETCD Configuration File on every etcd node:

For configuring etcd with our custom configuration it required to create configuration file in “/etc/etcd.env” location because we are calling this environment file in etcd service file.

Etcd configuration file for etcd node1:

[[email protected] etcd]# vi /etc/etcd.env

ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.45:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.45:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.45:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.45:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380
#ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-01.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-01-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-01.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-01-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true

Etcd configuration file for etcd node2:

[[email protected] etcd]# vi /etc/etcd.env

ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.46:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.46:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.46:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.46:2380
ETCD_NAME=etcd2
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380
#ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.46:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-02.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-02-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-02.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-02-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true

Etcd configuration file for etcd node3:

[[email protected]* etcd]# vi /etc/etcd.env

ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.47:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.47:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.47:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.47:2380
ETCD_NAME=etcd3
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-03.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-03-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-03.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-03-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true

Applying ETCD configuration on systemd for every etcd nodes:

In systemd based system it is required to reload daemon on every etcd nodes after service file change.

[[email protected]* etcd]# systemctl daemon-reload

Start and enable ETCD service on every:

Now we can start our etcd service by systemd command, It is also required to enable daemon on system boot time so we will need not to start service every time manually on system startup.

[[email protected]* etcd]# systemctl start etcd
[[email protected]* etcd]# systemctl enable etcd

Verifying ETCD cluster status:

For verifying etcd cluster status we can run below command on any etcd nodes.

[[email protected] ~]# etcdctl -C https://192.168.43.45:2379 --ca-file /etc/ssl/etcd/ssl/ca.pem cluster-health
member 649628565489a99c is healthy: got healthy result from https://192.168.43.45:2379
member caa56683e6af0137 is healthy: got healthy result from https://192.168.43.46:2379
member dc4795c6ff3e6627 is healthy: got healthy result from https://192.168.43.47:2379
cluster is healthy

If cluster is showing healthy that indicated we have configured everything properly so it’s time to use etcd based data store in our kubernetes cluster . In our future tutorials we will demonstrate how can we setup highly secure kubernetes cluster.


Leave a comment

Your email address will not be published. Required fields are marked *

4 thoughts on “Kubernetes ETCD Data Store Cluster with TLS Certificate.

  • Tony

    Thanks, This post is very useful, For a log time I was searching for secure ETCD configuration but didn’t find any good document. I was setting up ETCD with this configuration and it toke only 30 minutes , Great thanks again.

  • hos7ein

    tnx for this post

    I setup 3 node cluster by this article and after Verifying ETCD cluster status,show this warning :

    [[email protected] ~]# etcdctl -C https://10.10.22.184:2379 –ca-file /etc/ssl/etcd/ssl/ca.pem cluster-health
    2017-07-26 16:16:58.089694 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
    2017-07-26 16:16:58.089879 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
    member 36ecf80e03999cd is healthy: got healthy result from https://10.10.22.184:2379
    member 5c8ac3decbe0f5b7 is healthy: got healthy result from https://10.10.22.182:2379
    member fefe08e2b0a6f06d is healthy: got healthy result from https://10.10.22.183:2379
    cluster is healthy

    what’s the problem?

    tnx

    • admin Post author

      Apologies for delayed response, I was busy in official work, It seems you have missed out client IP somewhere in your openssl.conf file. Kindly verify all IPs again.