Howto – Set up a highly available instance of Rancher

Rancher is an open source software platform that enables organizations to run and manage Docker and Kubernetes in production. With Rancher, organizations no longer have to build a container services platform from scratch using a distinct set of open source technologies. Rancher supplies the entire software stack needed to manage containers in production.

Abstract:

In this guide we will be setting up a highly available instance of Rancher. The whole setup consists of four Debian Stretch servers, three on which docker is installed, and a fourth one which will perform the load balancing across the three Rancher servers. The whole rancher installation will be deployed containerized, and haproxy will serve as the load balancer.

Set-Up docker on Ubuntu:

Update the apt package index:

$ sudo apt-get update 

Install packages to allow apt to use a repository over HTTPS:

$ sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
 
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -  

$sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" 

$ sudo apt-get update

$ sudo apt-get install docker-ce=17.03.3~ce-0~ubuntu-xenial  

Install Prerequisites:

Install Kubectl:

$ sudo apt-get update && sudo apt-get install -y apt-transport-https
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo $ tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install -y kubectl 

Install rke:

$ wget https://github.com/rancher/rke/releases/download/v0.1.11/rke_linux-amd64
$ mv rke_linux-amd64 rke
$ mv rke /usr/bin/
$ chmod +x /usr/bin/rke
$ rke --version 

Configure single sign on:

(needs to be done on the first server only)

under the adm account:

$ ssh-keygen (no passphrase)
$ ssh-copy-id adm@rancher001.exampledomain.com
$ ssh-copy-id adm@rancher002.exampledomain.com
$ ssh-copy-id adm@rancher003.exampledomain.com 

test single sign on:

$ ssh adm@rancher001.exampledomain.com
$ ssh adm@rancher002.exampledomain.com
$ ssh adm@rancher003.exampledomain.com 

Create yml deployment file for our rancher cluster

under the adm user account:

$ cat < /home/adm/rancher-cluster.yml
nodes:
      address: rancher001.exampledomain.com user: adm role: [controlplane,worker,etcd]
      address: rancher002.exampledomain.com user: adm role: [controlplane,worker,etcd]
      address: rancher003.exampledomain.com user: adm role: [controlplane,worker,etcd] 
services:
      etcd:
      snapshot: true
      creation: 6h
      retention: 24h
EOF

add the adm account to the docker group (on all three hosts)

$ sudo usermod -aG docker adm 

(now first close your ssh session and log back in to apply the group membership change)

Install the kubernetes cluster:

$ sudo rke up --config ./rancher-cluster.yml 

The installer will have created a credential file for kubectl, need to copy this in the right place:

$ sudo chown adm:adm ~/kube_config_rancher-cluster.yml
$ mkdir ~/.kube
$ cp kube_config_rancher-cluster.yml /home/adm/.kube/config 

Test kubectl:

$ kubectl get all 

install helm & tiller

$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz
$ tar -xvf helm-v2.11.0-linux-amd64.tar.gz
$ cd linux-amd64/
$ sudo cp helm /usr/local/bin
$ sudo cp tiller /usr/local/bin
$ sudo chmod 755 /usr/local/bin/helm
$ sudo chmod 755 /usr/local/bin/tiller
$ kubectl -n kube-system create serviceaccount tiller
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller 

Test your tiller installation:

$ kubectl -n kube-system rollout status deploy/tiller-deploy 

And run the following command to validate Helm can talk to the tiller service:

$ helm version 

Install Rancher

Add the rancher stable repository to helm package manager:

$ helm repo add rancher-stable  https://releases.rancher.com/server-charts/stable 

Install rancher:

helm install rancher-stable/rancher --name rancher --namespace cattle-system --set hostname=rancherprov.exampledomain.com --set ingress.tls.source=secret --set privateCA=true 

Now create and sign a server certificate for our installation:

$ openssl genrsa -out ./rancherprov.exampledomain.com.key 2048
$ openssl req -new -sha256 -key ./rancherprov.exampledomain.com.key -out ./rancherprov.exampledomain.com.csr
     BE
     SomeLocation
     SomeTown
     SomeCompany
     IT
     rancherprov.exampledomain.com
     support@exampledomain.com
(no challenge password or optional company name, just hit enter)
cat rancherprov.exampledomain.com.csr 

Sign this certificate signing request on our own certificate authority:

In this case the request was signed on a Microsoft Certificate Authority.

Open a browser and navigate to your certificate authority: https://certificates.exampledomain.com/certsrv

• request a certificate
• advanced certificate request
• paste the certificate signing request
• select webserver template
• paste in attributes: san:dns=rancherprov.exampledomain.com
• submit
• select base64
• select download certificate

Paste the contents from the certificate signing request file:
rancherprov.exampledomain.com.crt in the signature box 

Add the certificate and key to the kubernetes cluster:

$ kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=rancherprov.exampledomain.com.crt --key=rancherprov.exampledomain.com.key 

Copy the certificate of our internal certificate authority to a file: cacerts.pem

Now install our certificate authority certificate in the kubernetes cluster:

$ kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem 

Upgrade Rancher

$ helm upgrade rancher rancher-stable /rancher \
  --set hostname=rancherprov.exampledomain.com \
  --set ingress.tls.source=secret \
  --set privateCA=true 

Install a load balancer to load balance the three rancher servers:

Installing HAProxy

It’s easy:

$ sudo apt install -y haproxy 

Configuring HAProxy

The haproxy configuration file can be found here: /etc/haproxy/haproxy.cfg

$ sudo vi /etc/haproxy/haproxy.cfg 

After modifying the configuration file, it’s required to restart the HAProxy service

$ sudo systemctl restart haproxy 

Example configuration:

This example will load balance http and https connections to three different backend servers.   There is no ssl offloading in this exaple by haproxy, rather, the ssl connections are using tcp load balancing with a sticky table.

global
log /dev/log    local0
log /dev/log    local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations     ca-base /etc/ssl/certs     crt-base /etc/ssl/private     # Default ciphers to use on SSL-enabled listening sockets.     # For more information, see ciphers(1SSL). This list is from:     #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/     # An alternative list with additional directives can be obtained from     #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy     ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS     ssl-default-bind-options no-sslv3
 defaults
 log     global
 mode    http
 option  httplog
 option  dontlognull
 timeout connect 5000
 timeout client  50000
 timeout server  50000
 errorfile 400 /etc/haproxy/errors/400.http
 errorfile 403 /etc/haproxy/errors/403.http
 errorfile 408 /etc/haproxy/errors/408.http
 errorfile 500 /etc/haproxy/errors/500.http
 errorfile 502 /etc/haproxy/errors/502.http
 errorfile 503 /etc/haproxy/errors/503.http
 errorfile 504 /etc/haproxy/errors/504.http
 frontend rancher_http
 bind *:80
 mode http
 default_backend rancher_http_backendnodes
 frontend rancher_https
 bind *:443
 mode tcp
 default_backend rancher_https_backendnodes
 backend rancher_http_backendnodes
 mode http
 balance roundrobin
 option forwardfor
 http-request set-header X-Forwarded-Port %[dst_port]
 http-request add-header X-Forwarded-Proto https if { ssl_fc }
 option httpchk HEAD / HTTP/1.1\r\nHost:\ rancher.exampledomain.com
 server node1 10.1.1.160:80 check
 server node2 10.1.1.161:80 check
 server node3 10.1.1.162:80 check
 backend rancher_https_backendnodes
 mode tcp
 balance roundrobin
 stick-table type ip size 200k expire 30m
 stick on src
 default-server inter 1s
 server node1 10.1.1.160:443 check
 server node2 10.1.1.161:443 check
 server node3 10.1.1.162:443 check
 listen stats
 bind :32700
 stats enable
 stats uri /
 stats hide-version
 stats auth admin:mypassword