Gitlab Runner and Docker in Docker dind configuration

About Docker + TLS

Docker client-server communication

For Docker-in-Docker (docker:*dind) services, we need to share the “client certificates directory” (with all docker containers).

Since 19.03, the docker-in-docker containers enable TLS by default.

The container startup generates all the required certificates (CA, server and client).

The docker (client) container probes for client certificates and enables TLS accordingly.

https://github.com/docker-library/docker/blob/master/dockerd-entrypoint.sh

How is TLS probed on the client?

When the “${DOCKER_TLS_CERTDIR}/client” contains the files ca.pem, cert.pem and client.key, the ‘docker:latest’ container will enable TLS by default. If the files are not available TLS will be disabled.
You can also force the use of TLS by defining DOCKER_TLS_VERIFY=1

In either case, the DOCKER_HOST will be defined accordingly. When TLS is enabled, it will be set to tcp://docker:2376. When TLS is disabled, it will be set to tcp://docker:2375.

Where is the data stored?

The folder where the daemon stores the certificates (CA, client, server) can be defined (overridden) by the DOCKER_CERT_PATH environment variable. The client certificates will be stored to ${DOCKER_TLS_DIR}/client.

The folder where the client reads the (client) certificates are read can be defined (overridden) by the DOCKER_CERT_PATH environment variable. It defaults to ${DOCKER_TLS_DIR}/client.

Please note that if ${DOCKER_CERT_PATH} is not equal to ${DOCKER_TLS_DIR}/client, you may need to force TLS by yourself.

Which means that you need to define the environment variables:

DOCKER_HOST = tcp://docker:2376
(default to tcp://docker:2375 otherwise)
DOCKER_TLS_VERIFY = 1

Docker server-registry communication

Note: This section may also be relevant for client-server and or client-registry communication

Docker (daemon) looks under /etc/docker/certs.d/<hostname>:<port> for docker registry certificates.

So we must map our company CA certificate for the GitLab registry (mygitserver.mydomain.com:4567) to that path.

GitLab Runner Installation

Prepare container configuration

Configuration directory for gitlab-runner

$ sudo mkdir -m 0700 -p /srv/docker-runner/config

(optional) Install Company CA certificate

Note: This is only required when mygitserver.mydomain .com uses a server certificate signed by a self-signed company CA.

For gitlab-runner

$ sudo mkdir -m 0700 /srv/docker-runner/config/certs 
 
$ sudo cp /usr/share/ca-certificates/mydomain.com/mycaserver.mydomain.com.crt /srv/docker-runner/config/certs/ 
 
$ sudo chmod 0600 /srv/docker-runner/config/certs/ mycaserver.mydomain.com.crt

For docker-in-docker
See explanation under (optional) with Mycompany CA certificate

$ sudo mkdir -m 0700 -p /srv/docker/certs.d/gitlab.mydomain.com\:4567  

$ sudo cp /usr/share/ca-certificates/mydomain.com/mycaserver.mydomain.com.crt /srv/docker/certs.d/mygitserver.mydomain.com\:4567/  

$ sudo chmod 0600 /srv/docker/certs.d/ mygitserver.mydomain.com \:4567/ mycaserver.mydomain.com.crt

Register the runner

Basic command

Command we use for registering docker runner (as Group Runner):

We define a ‘docker’ runner:

Needs to run in privileged mode
TLS verification is enabled
Default docker image (if not overridden in .gitlab-ci.yml) is debian:stretch-slim
Use better performing storage driver (overlay2; instead of vfs)
(see also Why use overlay2 as Docker storage driver?)
Share the “/cache” volume (TODO: why ?)
Share the “client certificates directory” (“/certs/client”) directory (see also About Docker + TLS why we do so)

$ docker run --rm -t -i \  
-v /srv/docker-runner/config:/etc/gitlab-runner \  
gitlab/gitlab-runner register \  
--non-interactive \  
--url 'https://mygitserver.mydomain.com/' \  
--registration-token '******' \  
--description 'docker-runner' \  
--locked='true' \  
--tag-list 'docker' \  
--executor 'docker' \  
--env 'DOCKER_DRIVER=overlay2' \  
--docker-privileged \  
--docker-tlsverify \  
--docker-image 'debian:stretch-slim' \  
--docker-volumes '/cache' \  
--docker-volumes '/certs/client'

(optional) with Company CA certificate

Note: This is only required when mygitserver.mydomain.com uses a server certificate signed by a self-signed company CA.

Direct volume mapping /srv/gitlab-runner/config/certs to /etc/docker/certs.d/ mygitserver.mydomain .com:4567 does not work (because of the “:” in “/etc/docker/certs.d/ mygitserver.mydomain .com:4567″).

So we map the parent directory: certs.d.

$ docker run --rm -t -i -v /srv/docker-runner/config:/etc/gitlab-runner \  
gitlab/gitlab-runner register \  
--non-interactive \  
--url 'https://mygitserver.mydomain.com/' \  
--tls-ca-file '/etc/docker-runner/certs/intermediate.mydomain.com.crt' \  
--registration-token '******' \  
--description 'docker-runner' \  
--locked='true' \  
--tag-list 'docker' \ 
--executor 'docker' \  
--env 'DOCKER_DRIVER=overlay2' \  
--docker-privileged \  
--docker-tlsverify \  
--docker-image 'debian:stretch-slim' \  
--docker-volumes '/cache' \  
--docker-volumes '/srv/docker/certs.d:/etc/docker/certs.d:ro' \  
--docker-volumes '/certs/client'

Example run

$ sudo -u myuser docker run --rm -t -i \  
 -v /srv/docker-runner/config:/etc/gitlab-runner \  
 gitlab/gitlab-runner register \  
 --non-interactive \  
 --url 'https://mygitserver.mydomain.com/' \  
 --registration-token '*****' \  
 --description 'docker-runner' \  
 --locked='true' \  
 --tag-list 'docker' \  
 --executor 'docker' \  
 --env 'DOCKER_DRIVER=overlay2' \  
 --docker-privileged \  
 --docker-tlsverify \  
 --docker-image 'debian:stretch-slim' \  
 --docker-volumes '/cache' \  
 --docker-volumes '/certs/client'

The output:

Runtime platform                                    arch=amd64 os=linux pid=7 revision=d0b76032 version=12.0.2  

Running in system-mode. 

Registering runner... succeeded                     runner=******   

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Review the configuration to support Company CA in Docker-in-Docker:

Eventually, update the concurrency to the allowed number of parallel jobs

$ sudo cat /srv/docker-runner/config/config.toml

 concurrent = 4  
 check_interval = 0 

 [session_server]  
 session_timeout = 1800 
 
 [[runners]]  
 name = "docker-runner"  
 url = "https://mygitserver.mydomain.com/"  
 token = "********"  
 executor = "docker"  
 environment = ["DOCKER_DRIVER=overlay2"]  

 [runners.custom_build_dir]  

 [runners.docker]  
 tls_verify = true  
 image = "debian:stretch-slim"  
 privileged = true  
 disable_entrypoint_overwrite = false  
 oom_kill_disable = false  
 disable_cache = false  
 volumes = ["/cache", "/certs/client"]  
 shm_size = 0  

 [runners.cache]  
 
 [runners.cache.s3]  

 [runners.cache.gcs]  
 
 [runners.custom]  
 run_exec = ""

Create the container :

$ sudo docker run \  
 --detach \  
 --name docker-runner \  
 --restart always \  
 -v /srv/docker-runner/config:/etc/gitlab-runner \  
 -v /var/run/docker.sock:/var/run/docker.sock \  
 gitlab/gitlab-runner:latest

Output:

c6c600202cd86098e0e01fb2433d4f51ed570444ac89234382bb089e01e7ef11

Use the container:

https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#use-docker-in-docker-executor
(2019/08/14) to-be-merged documentation: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31051/diffs

Why use overlay2 as Docker storage driver?

See https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#using-the-overlayfs-driver

Examples:

For example: build build server image(s):

https://mygitserver.mydomain.com/testproject/blob/master/.gitlab-ci.yml

This file is a template, and might need editing before it works on your project.
 Official docker image.
 image: docker:stable 
 XXX - Hmm, does not work using default services? (mygitserver.mydomain.com:4567/testproject)
 #
 Only required when the gitlab-runner does not provide the services itself in its config.toml file:
 [[runners]]
 [runners.docker]
 services = [ "docker:stable-dind" ]
 #
 services:
 name: docker:stable-dind
 # No longer required
 # command:
 #   - /bin/sh
 #   - -c
 #   - wget http://mycaserver.mydomain.com/mycacertificate.crt
 #         -O /usr/local/share/ca-certificates/mycacertificate.mydomain.com.pem
 #     && update-ca-certificates
 #     && dockerd-entrypoint.sh
 #     || exit 
 #
 See also
 https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#container-registry-examples
 for a nicely structure test/release/deploy script.
 #
 variables:
   # Force using TCP port 2376 (instead of default 2375)
   # see also
   # - https://github.com/docker-library/docs/tree/master/docker#tls
   # - https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.gitlab-ci.yml
   # XXX - done by default in docker:stable (19.03)
 DOCKER_HOST: tcp://docker:2376
 # DOCKER_TLS: 1
 DOCKER_TLS_VERIFY: 1
 # DOCKER_CERT_PATH: ~/.docker
 #   DOCKER_TLS_CERTDIR: "/certs"
 DOCKER_CERT_PATH: "/certs/client"
 #
   # See also
   # - https://gitlab.com/gitlab-org/gitlab-runner/issues/4501
   # - https://gitlab.com/gitlab-org/gitlab-runner/issues/4512
   # - https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31051/diffs
   # - https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#use-docker-in-docker-executor
   #
   # When using dind service we need to instruct docker, to talk with
   # the daemon started inside of the service. The daemon is
   # available with a network connection instead of the default
   # /var/run/docker.sock socket. docker:19.03-dind does this
   # automatically by setting the DOCKER_HOST in
   # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03/docker-entrypoint.sh#L23-L29
   #
   # The 'docker' hostname is the alias of the service container as described at
   # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services.
   #
   # When using dind, it's wise to use the overlayfs driver for
   # improved performance.
 Now fine by using volume_driver = "overlay2" in "config.toml"?
 DOCKER_DRIVER: overlay2
 # Specify to Docker where to create the certificates, Docker will
   # create them automatically on boot, and will create
   # /certs/client that will be shared between the service and
   # build container.
   #
   # XXX - Seems to defined by default in gitlab-runner (12.*) ?
 DOCKER_TLS_CERTDIR: "/certs"
 #
   CONTAINER_BASE_TEST_IMAGE: $CI_REGISTRY_IMAGE/base:$CI_COMMIT_REF_SLUG
   CONTAINER_BASE_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/base:latest
   CONTAINER_DEFAULT_TEST_IMAGE: $CI_REGISTRY_IMAGE/default:$CI_COMMIT_REF_SLUG
   CONTAINER_DEFAULT_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/default:latest
   # Default to UID/GID 1001/1001 since the user myuser@myserver has those IDs
   # If we define them otherwise we get issues when using sbuild.
   # E.g.
   #     sbuild --version
   # returns
   #     Can't get passwd entry for uid 1001:  at /usr/share/perl5/Sbuild/ConfBase.pm line 92.
   SBUILD_USER_ID: 1001
   SBUILD_GROUP_ID: 1001
   CONTAINER_SBUILD_TEST_IMAGE: $CI_REGISTRY_IMAGE/sbuild:$CI_COMMIT_REF_SLUG
   CONTAINER_SBUILD_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/sbuild:latest
 before_script:
 printenv
 ls -lhARw0 --color /etc/docker* || true
 ls -lhARw0 --color "${DOCKER_CERT_PATH:-${HOME}/.docker}" || true 
 - docker --log-level=debug --debug=true version
 - docker --log-level=debug --debug=true info
 docker --log-level=debug --debug=true login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY 
 stages:
 build-base
 build 
 build-master:info:
   stage: build-base
   tags:
     - docker
   script:
     - id
     - docker version
     - docker info
   only:
     - master
 build-master:base:
   stage: build-base
   tags:
     - docker
   script:
     - docker build --pull
         --tag="$CONTAINER_BASE_RELEASE_IMAGE"
         --file Dockerfile.buildserver-debian-base .
     - docker push "$CONTAINER_BASE_RELEASE_IMAGE"
   only:
     - master
 build-master:default:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build-master:base
   script:
     - docker build --pull
         --tag="$CONTAINER_DEFAULT_RELEASE_IMAGE"
         --file Dockerfile.buildserver-debian-9 .
     - docker push "$CONTAINER_DEFAULT_RELEASE_IMAGE"
   only:
     - master
 build-master:sbuild:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build-master:base
   script:
     - docker build --pull
         --tag="$CONTAINER_SBUILD_RELEASE_IMAGE"
         --build-arg user_id="${SBUILD_USER_ID}"
         --build-arg group_id="${SBUILD_GROUP_ID}"
         --file Dockerfile.buildserver-debian-9-sbuild .
     - docker push "$CONTAINER_SBUILD_RELEASE_IMAGE"
   only:
     - master
 build:base:
   stage: build-base
   tags:
     - docker
   script:
     - docker build --pull
         --tag="$CONTAINER_BASE_TEST_IMAGE"
         --file Dockerfile.buildserver-debian-base .
     - docker push "$CONTAINER_BASE_TEST_IMAGE"
   except:
     - master
 build:default:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build:base
   script:
     - docker build --pull
         --tag="$CONTAINER_DEFAULT_TEST_IMAGE"
         --file Dockerfile.buildserver-debian-9 .
     - docker push "$CONTAINER_DEFAULT_TEST_IMAGE"
   except:
     - master
 build:sbuild:
   stage: build
   tags:
     - docker
   # Fine because of ordering in 'stages' definition:
   # dependencies:
   #   - build:base
   script:
     - docker build --pull
         --tag="$CONTAINER_SBUILD_TEST_IMAGE"
         --build-arg user_id="${SBUILD_USER_ID}"
         --build-arg group_id="${SBUILD_GROUP_ID}"
         --file Dockerfile.buildserver-debian-9-sbuild .
     - docker push "$CONTAINER_SBUILD_TEST_IMAGE"
   except:
     - master

Building Docker containers for the GitLab Docker Registry

https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#container-registry-examples

GitLab runner build configuration: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.gitlab-ci.yml
buildserver build configuration: https://mygitserver.mydomain.com/products/buildserver

References

The Docker Executor:
- https://docs.gitlab.com/runner/executors/docker.html
GitLab Docker services:
- https://docs.gitlab.com/ce/ci/yaml/README.html#services
- https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-a-service
Docker-in-Docker Executor
- https://mygitserver.mydomain.com/help/ci/docker/using_docker_build.md#use-docker-in-docker-executor
GitLab runner:
- Overview:
  - https://docs.gitlab.com/runner/
  - https://docs.gitlab.com/ee/ci/runners/README.html
- Project pages
  - https://gitlab.com/gitlab-org/gitlab-runner
  - Build configuration: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.gitlab-ci.yml
  - Issue tracker: https://gitlab.com/gitlab-org/gitlab-runner/issues
- Install
  - As Docker Service: https://docs.gitlab.com/runner/install/docker.html
- Registering
  - https://docs.gitlab.com/runner/register/
- Commands
  - https://docs.gitlab.com/runner/commands/README.html
  - Configuration file: https://docs.gitlab.com/runner/commands/README.html#configuration-file
- Configuration
  - https://docs.gitlab.com/runner/configuration/index.html
  - Self-signed certificates or custom Certification Authorities
    - https://docs.gitlab.com/runner/configuration/tls-self-signed.html
    - Location of company CA (for docker daemon)
      - https://docs.docker.com/engine/security/certificates/#understand-the-configuration
    - Using TLS in Docker 19.03+
      - https://github.com/docker-library/docs/tree/master/docker#tls
    - Nice trick to get the certificate from the host itself using openssl s_client:
      http://germanylandofinnovation.com/questions/52438/gitlab-ci-runner-ignoriert-selbstsigniertes-zertifikat
  - Advanced configuration (config.toml)
    - TOML format: https://github.com/toml-lang/toml
    - https://docs.gitlab.com/runner/configuration/advanced-configuration.html
    - What’s the use of the [runners.docker] section in config.toml for use case with docker machine?
      https://stackoverflow.com/questions/47814564/whats-the-use-of-the-runners-docker-section-in-config-toml-for-use-case-with
  - Example from gitlab-runner
    - https://gitlab.com/gitlab-org/gitlab-runner/blob/master/config.toml.example
- Runner Executors
  - https://docs.gitlab.com/runner/executors/README.html

Gitlab Runner and Docker in Docker dind configuration

About Docker + TLS

Docker client-server communication

How is TLS probed on the client?

Where is the data stored?

Docker server-registry communication

GitLab Runner Installation

Prepare container configuration

Register the runner

Why use overlay2 as Docker storage driver?

Published by

Ronny Van den Broeck

Leave a Reply Cancel reply

About Docker + TLS

Docker client-server communication

How is TLS probed on the client?

Where is the data stored?

Docker server-registry communication

GitLab Runner Installation

Prepare container configuration

Register the runner

Why use overlay2 as Docker storage driver?

Dit delen:

Like this:

Related

Published by

Ronny Van den Broeck

Leave a Reply Cancel reply