Gitlab Runner and Docker in Docker dind configuration

About Docker + TLS 

Docker client-server communication 

For Docker-in-Docker (docker:*dind) services, we need to share the “client certificates directory” (with all docker containers). 

Since 19.03, the docker-in-docker containers enable TLS by default. 

The container startup generates all the required certificates (CA, server and client). 

The docker (client) container probes for client certificates and enables TLS accordingly. 

How is TLS probed on the client? 

  1. When the “${DOCKER_TLS_CERTDIR}/client” contains the files ca.pem, cert.pem and client.key, the ‘docker:latest’ container will enable TLS by default. If the files are not available TLS will be disabled. 
  2. You can also force the use of TLS by defining DOCKER_TLS_VERIFY=1 

In either case, the DOCKER_HOST will be defined accordingly. When TLS is enabled, it will be set to tcp://docker:2376. When TLS is disabled, it will be set to tcp://docker:2375. 

Where is the data stored? 

The folder where the daemon stores the certificates (CA, client, server) can be defined (overridden) by the DOCKER_CERT_PATH environment variable. The client certificates will be stored to ${DOCKER_TLS_DIR}/client. 

The folder where the client reads the (client) certificates are read can be defined (overridden) by the DOCKER_CERT_PATH environment variable. It defaults to ${DOCKER_TLS_DIR}/client. 

Please note that if ${DOCKER_CERT_PATH} is not equal to ${DOCKER_TLS_DIR}/client, you may need to force TLS by yourself

Which means that you need to define the environment variables: 

  • DOCKER_HOST = tcp://docker:2376 
    (default to tcp://docker:2375 otherwise) 
  • DOCKER_TLS_VERIFY = 1 

Docker server-registry communication 

NoteThis section may also be relevant for client-server and or client-registry communication 

Docker (daemon) looks under /etc/docker/certs.d/<hostname>:<port> for docker registry certificates. 

See also https://docs.docker.com/engine/security/certificates/#understand-the-configuration

So we must map our company CA certificate for the GitLab registry (mygitserver.mydomain.com:4567) to that path. 

GitLab Runner Installation 

Prepare container configuration 

Configuration directory for gitlab-runner 

$ sudo mkdir -m 0700 -p /srv/docker-runner/config  

(optional) Install Company CA certificate 

Note: This is only required when mygitserver.mydomain .com uses a server certificate signed by a self-signed company CA. 

  1. For gitlab-runner 
$ sudo mkdir -m 0700 /srv/docker-runner/config/certs 
 
$ sudo cp /usr/share/ca-certificates/mydomain.com/mycaserver.mydomain.com.crt /srv/docker-runner/config/certs/ 
 
$ sudo chmod 0600 /srv/docker-runner/config/certs/ mycaserver.mydomain.com.crt   
  1. For docker-in-docker 
    See explanation under (optional) with Mycompany CA certificate 
$ sudo mkdir -m 0700 -p /srv/docker/certs.d/gitlab.mydomain.com\:4567  

$ sudo cp /usr/share/ca-certificates/mydomain.com/mycaserver.mydomain.com.crt /srv/docker/certs.d/mygitserver.mydomain.com\:4567/  

$ sudo chmod 0600 /srv/docker/certs.d/ mygitserver.mydomain.com \:4567/ mycaserver.mydomain.com.crt   

Register the runner 

Basic command 

Command we use for registering docker runner (as Group Runner): 

We define a ‘docker’ runner: 

  • Needs to run in privileged mode 
  • TLS verification is enabled 
  • Default docker image (if not overridden in .gitlab-ci.yml) is debian:stretch-slim 
  • Use better performing storage driver (overlay2; instead of vfs) 
    (see also Why use overlay2 as Docker storage driver?
  • Share the “/cache” volume (TODO: why ?
  • Share the “client certificates directory” (“/certs/client”) directory (see also About Docker + TLS why we do so) 
$ docker run --rm -t -i \  
-v /srv/docker-runner/config:/etc/gitlab-runner \  
gitlab/gitlab-runner register \  
--non-interactive \  
--url 'https://mygitserver.mydomain.com/' \  
--registration-token '******' \  
--description 'docker-runner' \  
--locked='true' \  
--tag-list 'docker' \  
--executor 'docker' \  
--env 'DOCKER_DRIVER=overlay2' \  
--docker-privileged \  
--docker-tlsverify \  
--docker-image 'debian:stretch-slim' \  
--docker-volumes '/cache' \  
--docker-volumes '/certs/client'  

(optional) with Company CA certificate 

Note: This is only required when mygitserver.mydomain.com uses a server certificate signed by a self-signed company CA. 

Direct volume mapping /srv/gitlab-runner/config/certs to /etc/docker/certs.d/ mygitserver.mydomain .com:4567 does not work (because of the “:” in “/etc/docker/certs.d/ mygitserver.mydomain .com:4567″). 

So we map the parent directory: certs.d

$ docker run --rm -t -i -v /srv/docker-runner/config:/etc/gitlab-runner \  
gitlab/gitlab-runner register \  
--non-interactive \  
--url 'https://mygitserver.mydomain.com/' \  
--tls-ca-file '/etc/docker-runner/certs/intermediate.mydomain.com.crt' \  
--registration-token '******' \  
--description 'docker-runner' \  
--locked='true' \  
--tag-list 'docker' \ 
--executor 'docker' \  
--env 'DOCKER_DRIVER=overlay2' \  
--docker-privileged \  
--docker-tlsverify \  
--docker-image 'debian:stretch-slim' \  
--docker-volumes '/cache' \  
--docker-volumes '/srv/docker/certs.d:/etc/docker/certs.d:ro' \  
--docker-volumes '/certs/client'  

Example run 

$ sudo -u myuser docker run --rm -t -i \  
 -v /srv/docker-runner/config:/etc/gitlab-runner \  
 gitlab/gitlab-runner register \  
 --non-interactive \  
 --url 'https://mygitserver.mydomain.com/' \  
 --registration-token '*****' \  
 --description 'docker-runner' \  
 --locked='true' \  
 --tag-list 'docker' \  
 --executor 'docker' \  
 --env 'DOCKER_DRIVER=overlay2' \  
 --docker-privileged \  
 --docker-tlsverify \  
 --docker-image 'debian:stretch-slim' \  
 --docker-volumes '/cache' \  
 --docker-volumes '/certs/client'  

The output:

Runtime platform                                    arch=amd64 os=linux pid=7 revision=d0b76032 version=12.0.2  

Running in system-mode. 

Registering runner... succeeded                     runner=******   

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!  

Review the configuration to support Company CA in Docker-in-Docker: 

Eventually, update the concurrency to the allowed number of parallel jobs 

$ sudo cat /srv/docker-runner/config/config.toml  
 concurrent = 4  
 check_interval = 0 

 [session_server]  
 session_timeout = 1800 
 
 [[runners]]  
 name = "docker-runner"  
 url = "https://mygitserver.mydomain.com/"  
 token = "********"  
 executor = "docker"  
 environment = ["DOCKER_DRIVER=overlay2"]  

 [runners.custom_build_dir]  

 [runners.docker]  
 tls_verify = true  
 image = "debian:stretch-slim"  
 privileged = true  
 disable_entrypoint_overwrite = false  
 oom_kill_disable = false  
 disable_cache = false  
 volumes = ["/cache", "/certs/client"]  
 shm_size = 0  

 [runners.cache]  
 
 [runners.cache.s3]  

 [runners.cache.gcs]  
 
 [runners.custom]  
 run_exec = ""  

Create the container :

$ sudo docker run \  
 --detach \  
 --name docker-runner \  
 --restart always \  
 -v /srv/docker-runner/config:/etc/gitlab-runner \  
 -v /var/run/docker.sock:/var/run/docker.sock \  
 gitlab/gitlab-runner:latest  

Output:

c6c600202cd86098e0e01fb2433d4f51ed570444ac89234382bb089e01e7ef11  

Use the container: 

Why use overlay2 as Docker storage driver? 

See https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#using-the-overlayfs-driver 

Examples: 

For example: build build server image(s): 

https://mygitserver.mydomain.com/testproject/blob/master/.gitlab-ci.yml

This file is a template, and might need editing before it works on your project.
 Official docker image.
 image: docker:stable 
 XXX - Hmm, does not work using default services? (mygitserver.mydomain.com:4567/testproject)
 #
 Only required when the gitlab-runner does not provide the services itself in its config.toml file:
 [[runners]]
 [runners.docker]
 services = [ "docker:stable-dind" ]
 #
 services:
 name: docker:stable-dind
 # No longer required
 # command:
 #   - /bin/sh
 #   - -c
 #   - wget http://mycaserver.mydomain.com/mycacertificate.crt
 #         -O /usr/local/share/ca-certificates/mycacertificate.mydomain.com.pem
 #     && update-ca-certificates
 #     && dockerd-entrypoint.sh
 #     || exit 
 #
 See also
 https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#container-registry-examples
 for a nicely structure test/release/deploy script.
 #
 variables:
   # Force using TCP port 2376 (instead of default 2375)
   # see also
   # - https://github.com/docker-library/docs/tree/master/docker#tls
   # - https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.gitlab-ci.yml
   # XXX - done by default in docker:stable (19.03)
 DOCKER_HOST: tcp://docker:2376
 # DOCKER_TLS: 1
 DOCKER_TLS_VERIFY: 1
 # DOCKER_CERT_PATH: ~/.docker
 #   DOCKER_TLS_CERTDIR: "/certs"
 DOCKER_CERT_PATH: "/certs/client"
 #
   # See also
   # - https://gitlab.com/gitlab-org/gitlab-runner/issues/4501
   # - https://gitlab.com/gitlab-org/gitlab-runner/issues/4512
   # - https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31051/diffs
   # - https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#use-docker-in-docker-executor
   #
   # When using dind service we need to instruct docker, to talk with
   # the daemon started inside of the service. The daemon is
   # available with a network connection instead of the default
   # /var/run/docker.sock socket. docker:19.03-dind does this
   # automatically by setting the DOCKER_HOST in
   # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03/docker-entrypoint.sh#L23-L29
   #
   # The 'docker' hostname is the alias of the service container as described at
   # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services.
   #
   # When using dind, it's wise to use the overlayfs driver for
   # improved performance.
 Now fine by using volume_driver = "overlay2" in "config.toml"?
 DOCKER_DRIVER: overlay2
 # Specify to Docker where to create the certificates, Docker will
   # create them automatically on boot, and will create
   # /certs/client that will be shared between the service and
   # build container.
   #
   # XXX - Seems to defined by default in gitlab-runner (12.*) ?
 DOCKER_TLS_CERTDIR: "/certs"
 #
   CONTAINER_BASE_TEST_IMAGE: $CI_REGISTRY_IMAGE/base:$CI_COMMIT_REF_SLUG
   CONTAINER_BASE_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/base:latest
   CONTAINER_DEFAULT_TEST_IMAGE: $CI_REGISTRY_IMAGE/default:$CI_COMMIT_REF_SLUG
   CONTAINER_DEFAULT_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/default:latest
   # Default to UID/GID 1001/1001 since the user myuser@myserver has those IDs
   # If we define them otherwise we get issues when using sbuild.
   # E.g.
   #     sbuild --version
   # returns
   #     Can't get passwd entry for uid 1001:  at /usr/share/perl5/Sbuild/ConfBase.pm line 92.
   SBUILD_USER_ID: 1001
   SBUILD_GROUP_ID: 1001
   CONTAINER_SBUILD_TEST_IMAGE: $CI_REGISTRY_IMAGE/sbuild:$CI_COMMIT_REF_SLUG
   CONTAINER_SBUILD_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/sbuild:latest
 before_script:
 printenv
 ls -lhARw0 --color /etc/docker* || true
 ls -lhARw0 --color "${DOCKER_CERT_PATH:-${HOME}/.docker}" || true 
 - docker --log-level=debug --debug=true version
 - docker --log-level=debug --debug=true info
 docker --log-level=debug --debug=true login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY 
 stages:
 build-base
 build 
 build-master:info:
   stage: build-base
   tags:
     - docker
   script:
     - id
     - docker version
     - docker info
   only:
     - master
 build-master:base:
   stage: build-base
   tags:
     - docker
   script:
     - docker build --pull
         --tag="$CONTAINER_BASE_RELEASE_IMAGE"
         --file Dockerfile.buildserver-debian-base .
     - docker push "$CONTAINER_BASE_RELEASE_IMAGE"
   only:
     - master
 build-master:default:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build-master:base
   script:
     - docker build --pull
         --tag="$CONTAINER_DEFAULT_RELEASE_IMAGE"
         --file Dockerfile.buildserver-debian-9 .
     - docker push "$CONTAINER_DEFAULT_RELEASE_IMAGE"
   only:
     - master
 build-master:sbuild:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build-master:base
   script:
     - docker build --pull
         --tag="$CONTAINER_SBUILD_RELEASE_IMAGE"
         --build-arg user_id="${SBUILD_USER_ID}"
         --build-arg group_id="${SBUILD_GROUP_ID}"
         --file Dockerfile.buildserver-debian-9-sbuild .
     - docker push "$CONTAINER_SBUILD_RELEASE_IMAGE"
   only:
     - master
 build:base:
   stage: build-base
   tags:
     - docker
   script:
     - docker build --pull
         --tag="$CONTAINER_BASE_TEST_IMAGE"
         --file Dockerfile.buildserver-debian-base .
     - docker push "$CONTAINER_BASE_TEST_IMAGE"
   except:
     - master
 build:default:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build:base
   script:
     - docker build --pull
         --tag="$CONTAINER_DEFAULT_TEST_IMAGE"
         --file Dockerfile.buildserver-debian-9 .
     - docker push "$CONTAINER_DEFAULT_TEST_IMAGE"
   except:
     - master
 build:sbuild:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build:base
   script:
     - docker build --pull
         --tag="$CONTAINER_SBUILD_TEST_IMAGE"
         --build-arg user_id="${SBUILD_USER_ID}"
         --build-arg group_id="${SBUILD_GROUP_ID}"
         --file Dockerfile.buildserver-debian-9-sbuild .
     - docker push "$CONTAINER_SBUILD_TEST_IMAGE"
   except:
     - master

See also https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.gitlab-ci.yml 

Building Docker containers for the GitLab Docker Registry 

https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#container-registry-examples

References 

Published by

Ronny Van den Broeck

I'm a network and system engineer for more than 20 years now. During this period I became a pro in hunting down one's and zero's, with an eager mindset to help people accomplish the same or abstract them away from the matrix.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s