Running Gitlab runner in a container and using it to build and upload image

Running GitLab Runner in a Docker container can be a great way to isolate your build environment and easily deploy your runners to different machines. Here are the steps to set up GitLab Runner in a Docker container:

Run the gitlab runner in a container

1. Create a docker volume to store the gitlab runner config

   docker volume create gitlab_runner_config

2. Start the runner

   docker run -d --name gitlab-runner --restart always -v $XDG_RUNTIME_DIR/docker.sock:/var/run/docker.sock -v gitlab_runner_config:/etc/gitlab-runner gitlab/gitlab-runner:latest

   In the command shown above, the Docker socket is mounted to the container to enable the GitLab Runner to generate additional containers while executing the job. If you're using a rootful Docker, you can use /var/run/docker.sock, but since I'm using a rootless Docker, the XDG_RUNTIME_DIR is defined on my machine. Additionally, we attach a named volume that we created earlier, which will store the configurations that are automatically generated by the runner upon registration.

Register the gitlab runner

docker run --rm -it -v gitlab_runner_config:/etc/gitlab-runner gitlab/gitlab-runner:latest register -n \
 --url https://your.gitlab.url/ \
 --registration-token <GITLAB_TOKEN>\
 --executor docker \
 --description <Enter your description> \
 --docker-image "docker:23.0.3-cli" \
 --docker-privileged \
 --docker-volumes gitlab_certs_client:/certs/client \
 --tag-list <Enter your tag list>

• docker run: This starts a new Docker container.

• --rm: This option automatically removes the container after it exits.

• -it: This option allocates a pseudo-TTY and opens an interactive terminal.

• -v gitlab-runner-config:/etc/gitlab-runner: This mounts a named volume named "gitlab-runner-config" to the /etc/gitlab-runner directory in the container. This allows the container to access the GitLab Runner configuration file.

• gitlab/gitlab-runner:latest: This specifies the GitLab Runner Docker image to use.

• register: This tells the GitLab Runner to register itself with the GitLab server.

• -n: This option specifies that the runner should not perform any builds automatically after registration.

• --url https://your.gitlab.url/: This specifies the URL of the GitLab instance to register the runner with.

• --registration-token : This specifies the registration token to use. You can obtain this token from your GitLab instance.

• --executor docker: This specifies that the GitLab Runner should use Docker as its execution environment.

• --description : This allows you to enter a description for the runner.

• --docker-image "docker:23.0.3-cli": This specifies the Docker image to use for running the build.

• --docker-privileged: This enables privileged mode for the Docker container.

• --docker-volumes gitlab_certs_client:/certs/client: This mounts a named volume named "gitlab_certs_client" to the /certs/client directory in the container. This allows the service and build container to communicate with one another with TLS.

• --tag-list : This allows you to specify any tags to assign to the runner.

To build an docker image, here is an example of the .gitlab-ci.yml

variables:
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:23.0.3-dind

build:
  stage: build
  script:
    - docker build -t my-docker-image .
    - docker run my-docker-image /script/to/run/tests

To utilize the official Docker Hub registry for pulling and pushing images, simply modify the aforementioned example. For those, like myself, who employ a private Docker registry, additional configuration steps are necessary.

Configuration for Docker-in-Docker

As TLS is in use, it is essential to direct Docker to employ the correct CA certificate when connecting to the private registry; failing to do so may result in an x509: certificate signed by unknown authority error.

Let\'s first create a docker volume to hold the docker daemon configurations for the dind container.

docker volume create gitlab_runner_docker_config

Next, navigate to the newly created directory and create a daemon.json file with the following content.

{
  "registry-mirrors": [
    "https://registry-mirror.example.com:28000"
  ]
}

Within the same directory create a directory in this format cert.d/<registry.url>:<port> to hold the CA certificate.

mkdir -p cert.d/registry-mirror.example.com:28000

Next, copy your CA certificate to the newly created directory.

Proceed to update the config.toml file located in the gitlab_runner_config directory. Within config.toml, add the gitlab_runner_docker_config to the volumes parameter.

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab runner docker executor"
  url = "https://your.gitlab.url/"
  id = 1
  token = "TOKEN"
  token_obtained_at = 2023-04-17T05:42:11Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"

  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "docker:23.0.3-cli"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = [
      "gitlab_runner_docker_config:/etc/docker",
      "gitlab_certs_client:/certs/client",
      "/cache"
    ]
    shm_size = 0

Reload the gitlab runner configuration by executing the following command in your gitlab runner container

gitlab-runner restart

After implementing the changes, the updated Docker daemon configurations will be available within the Docker-in-Docker (dind) containers as they run. However, there is one more step: guiding the Docker daemon to log into the private registry.

build:
  before_script:
    - echo "$DOCKER_REGISTRY_PASS" | docker login $DOCKER_REGISTRY --username $DOCKER_REGISTRY_USER --password-stdin

Now you should be able to build, pull and push images within your gitlab runner using docker-in-docker image.

What if you are using a self-hosted gitlab?

If you are using a self-hosted gitlab like me, you will need update the CA certificate on the default docker job container and the dind service container.

Add the following pre_build_script into config.toml which will update the ca-certificate within the dind container. (Modify the script if you are using a Ubuntu based dind image)

pre_bulid_script = """
  apk update >/dev/null
  apk add ca-certificates > /dev/null
  cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
  update-ca-certificates --refresh > /dev/null
"""

Next, we need to ensure that the ca.crt is available at /etc/gitlab-runner/certs. To achieve that , we ill need to mount a new volume in config.toml

volumes = [
  ...
  "/path/to/your/ca.crt:/etc/gitlab-runner/certs/ca.crt:ro",
  ...
]

Your final config.toml may look like this.

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab runner docker executor"
  url = "https://your.gitlab.url/"
  id = 6
  token = "TOKEN"
  token_obtained_at = 2023-04-17T05:42:11Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  pre_build_script = """
    apk update >/dev/null
    apk add ca-certificates > /dev/null
    rm -rf /var/cache/apk/*
    cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
    update-ca-certificates --fresh > /dev/null
    mkdir -p /etc/docker/certs.d/registry-mirror.example.com:28000
    cp /etc/gitlab-runner/certs/ca.crt /etc/docker/certs.d/registry-mirror.example.com:28000/ca.crt
  """

  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "docker:23.0.3-cli"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = [
      "gitlab_runner_docker_config:/etc/docker",
      "gitlab_certs_client:/certs/client",
      "/home/docker/.docker/certs/internal-ca.crt:/etc/gitlab-runner/certs/ca.crt:ro",
      "/cache"
    ]
    shm_size = 0

Lastly, make sure that the CA cert is copied to your /path/to/docker/volumes/gitlab_runner_config/_data/certs/ and named as ca.crt.

Restart your gitlab runner and your pipeline should be working smoothly now.

References

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
https://docs.gitlab.com/ee/ci/yaml/
https://docs.gitlab.com/runner/configuration/tls-self-signed.html