Howto – Set up a highly available instance of Rancher

Rancher is an open source software platform that enables organizations to run and manage Docker and Kubernetes in production. With Rancher, organizations no longer have to build a container services platform from scratch using a distinct set of open source technologies. Rancher supplies the entire software stack needed to manage containers in production.

Abstract:

In this guide we will be setting up a highly available instance of Rancher. The whole setup consists of four Debian Stretch servers, three on which docker is installed, and a fourth one which will perform the load balancing across the three Rancher servers. The whole rancher installation will be deployed containerized, and haproxy will serve as the load balancer.

Set-Up docker on Ubuntu:

Update the apt package index:

$ sudo apt-get update 

Install packages to allow apt to use a repository over HTTPS:

$ sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

$ sudo apt-get update

$ sudo apt-get install docker-ce=17.03.3~ce-0~ubuntu-xenial

Install Prerequisites:

Install Kubectl:

$ sudo apt-get update && sudo apt-get install -y apt-transport-https
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo $ tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install -y kubectl

Install rke:

$ wget https://github.com/rancher/rke/releases/download/v0.1.11/rke_linux-amd64
$ mv rke_linux-amd64 rke
$ mv rke /usr/bin/
$ chmod +x /usr/bin/rke
$ rke --version

Configure single sign on:

(needs to be done on the first server only)

under the adm account:

$ ssh-keygen (no passphrase)
$ ssh-copy-id adm@rancher001.exampledomain.com
$ ssh-copy-id adm@rancher002.exampledomain.com
$ ssh-copy-id adm@rancher003.exampledomain.com

test single sign on:

$ ssh adm@rancher001.exampledomain.com
$ ssh adm@rancher002.exampledomain.com
$ ssh adm@rancher003.exampledomain.com

Create yml deployment file for our rancher cluster

under the adm user account:

$ cat < /home/adm/rancher-cluster.yml
nodes:
address: rancher001.exampledomain.com user: adm role: [controlplane,worker,etcd]
address: rancher002.exampledomain.com user: adm role: [controlplane,worker,etcd]
address: rancher003.exampledomain.com user: adm role: [controlplane,worker,etcd]
services:
etcd:
snapshot: true
creation: 6h
retention: 24h
EOF

add the adm account to the docker group (on all three hosts)

$ sudo usermod -aG docker adm 

(now first close your ssh session and log back in to apply the group membership change)

Install the kubernetes cluster:

$ sudo rke up --config ./rancher-cluster.yml 

The installer will have created a credential file for kubectl, need to copy this in the right place:

$ sudo chown adm:adm ~/kube_config_rancher-cluster.yml
$ mkdir ~/.kube
$ cp kube_config_rancher-cluster.yml /home/adm/.kube/config

Test kubectl:

$ kubectl get all 

install helm & tiller

$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz
$ tar -xvf helm-v2.11.0-linux-amd64.tar.gz
$ cd linux-amd64/
$ sudo cp helm /usr/local/bin
$ sudo cp tiller /usr/local/bin
$ sudo chmod 755 /usr/local/bin/helm
$ sudo chmod 755 /usr/local/bin/tiller
$ kubectl -n kube-system create serviceaccount tiller
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller

Test your tiller installation:

$ kubectl -n kube-system rollout status deploy/tiller-deploy 

And run the following command to validate Helm can talk to the tiller service:

$ helm version 

Install Rancher

Add the rancher stable repository to helm package manager:

$ helm repo add rancher-stable  https://releases.rancher.com/server-charts/stable 

Install rancher:

helm install rancher-stable/rancher --name rancher --namespace cattle-system --set hostname=rancherprov.exampledomain.com --set ingress.tls.source=secret --set privateCA=true 

Now create and sign a server certificate for our installation:

$ openssl genrsa -out ./rancherprov.exampledomain.com.key 2048
$ openssl req -new -sha256 -key ./rancherprov.exampledomain.com.key -out ./rancherprov.exampledomain.com.csr
BE
SomeLocation
SomeTown
SomeCompany
IT
rancherprov.exampledomain.com
support@exampledomain.com
(no challenge password or optional company name, just hit enter)
cat rancherprov.exampledomain.com.csr

Sign this certificate signing request on our own certificate authority:

In this case the request was signed on a Microsoft Certificate Authority.

Open a browser and navigate to your certificate authority: https://certificates.exampledomain.com/certsrv

  • request a certificate
  • advanced certificate request
  • paste the certificate signing request
  • select webserver template
  • paste in attributes: san:dns=rancherprov.exampledomain.com
  • submit
  • select base64
  • select download certificate

Paste the contents from the certificate signing request file:
rancherprov.exampledomain.com.crt in the signature box 

Add the certificate and key to the kubernetes cluster:

$ kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=rancherprov.exampledomain.com.crt --key=rancherprov.exampledomain.com.key 

Copy the certificate of our internal certificate authority to a file: cacerts.pem

Now install our certificate authority certificate in the kubernetes cluster:

$ kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem 

Upgrade Rancher

$ helm upgrade rancher rancher-stable /rancher \
  --set hostname=rancherprov.exampledomain.com \
  --set ingress.tls.source=secret \
  --set privateCA=true

Install a load balancer to load balance the three rancher servers:

Installing HAProxy

It’s easy:

$ sudo apt install -y haproxy 

Configuring HAProxy

The haproxy configuration file can be found here: /etc/haproxy/haproxy.cfg

$ sudo vi /etc/haproxy/haproxy.cfg 

After modifying the configuration file, it’s required to restart the HAProxy service

$ sudo systemctl restart haproxy 

Example configuration:

This example will load balance http and https connections to three different backend servers.   There is no ssl offloading in this exaple by haproxy, rather, the ssl connections are using tcp load balancing with a sticky table.

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend rancher_http
bind *:80
mode http
default_backend rancher_http_backendnodes
frontend rancher_https
bind *:443
mode tcp
default_backend rancher_https_backendnodes
backend rancher_http_backendnodes
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:\ rancher.exampledomain.com
server node1 10.1.1.160:80 check
server node2 10.1.1.161:80 check
server node3 10.1.1.162:80 check
backend rancher_https_backendnodes
mode tcp
balance roundrobin
stick-table type ip size 200k expire 30m
stick on src
default-server inter 1s
server node1 10.1.1.160:443 check
server node2 10.1.1.161:443 check
server node3 10.1.1.162:443 check
listen stats
bind :32700
stats enable
stats uri /
stats hide-version
stats auth admin:mypassword

Published by

Ronny Van den Broeck

I'm a network and system engineer for more than 20 years now. During this period I became a pro in hunting down one's and zero's, with an eager mindset to help people accomplish the same or abstract them away from the matrix.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s