GitLab CI: Runners, Executors and a Real Pipeline

GitLab CI/CD runs jobs defined in .gitlab-ci.yml on machines called runners. This post builds on the GitLab server from the previous post: it registers a Docker runner and writes a pipeline that builds, tests and deploys.

Versions in use on this date. GitLab 16.9 was released on 15 February 2024 and GitLab Runner 16.9.0 was tagged the same day. Docker Engine 25.0 shipped on 19 January 2024. Since GitLab 16.0 the old registration token flow is deprecated; the supported way is to create the runner in the UI and register it with an authentication token (it starts with glrt-).

Architecture

flowchart TB
  DEV[Developer push or merge request] --> GL[GitLab server]
  GL -->|job queue| API[Runner API]
  subgraph RH[Runner host - separate VM]
    RUN[gitlab-runner service<br/>concurrent 4] --> D[Docker Engine]
    D --> J1[Job container 1<br/>image from .gitlab-ci.yml]
    D --> J2[Job container 2]
    D --> SV[Service containers<br/>postgres redis]
  end
  RUN -->|long poll for jobs| API
  J1 -->|push image| REG[GitLab container registry]
  J1 -->|artifacts and cache| GL
  J2 -->|deploy over SSH or kubectl| TGT[Target environment]

A runner pulls jobs from GitLab (no inbound ports are needed on the runner), so it can sit in a private network. The runner host should not be the GitLab server: CI executes arbitrary code and can exhaust CPU, memory and disk.

Install and register

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt install -y gitlab-runner docker.io
sudo usermod -aG docker gitlab-runner

In GitLab, open Settings, CI/CD, Runners, New project runner (or an instance or group runner as an admin). Give it tags, copy the token, then:

sudo gitlab-runner register --non-interactive \
  --url https://gitlab.example.com --token glrt-xxxxxxxxxxxx \
  --executor docker --docker-image alpine:3.19 \
  --description "docker-runner-01"

Key settings in /etc/gitlab-runner/config.toml:

concurrent = 4
check_interval = 3

[[runners]]
  name = "docker-runner-01"
  executor = "docker"
  [runners.docker]
    image = "alpine:3.19"
    pull_policy = ["if-not-present"]
    privileged = false
    volumes = ["/cache"]
    shm_size = 0

Pick the executor deliberately:

Executor	Isolation	Use for
docker	Clean container per job	Most projects (default choice)
shell	None, runs as the runner user	Trusted jobs that need the host
kubernetes	Pod per job	Autoscaling on a cluster

A real pipeline

stages: [build, test, deploy]

variables:
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

build:
  stage: build
  image: docker:25.0
  services: ["docker:25.0-dind"]
  variables: { DOCKER_TLS_CERTDIR: "/certs" }
  tags: [docker]
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $IMAGE .
    - docker push $IMAGE

unit-test:
  stage: test
  image: node:20-alpine
  cache:
    key: { files: [package-lock.json] }
    paths: [node_modules/]
  script: [npm ci, npm test]
  artifacts:
    when: always
    reports: { junit: junit.xml }

deploy-staging:
  stage: deploy
  image: alpine:3.19
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  environment: { name: staging }
  script:
    - echo "deploy $IMAGE"

Docker-in-Docker (dind) needs privileged = true on that runner, which effectively gives jobs root on the host. Keep that on a dedicated, isolated runner for trusted projects only, or build with Kaniko or Buildah, which need no privileged mode. Do not mount /var/run/docker.sock into jobs on shared runners, because it hands out control of the host.

Operations and security

• Tags and protected runners: restrict deploy jobs to a runner marked protected, which runs only on protected branches and tags.
• Variables: store secrets under Settings, CI/CD, Variables as masked and protected; for stronger control use an external secret store.
• Caching: key the cache on the lockfile, as above. Artifacts pass files between stages; cache speeds up repeat installs.
• Disk: clean old images with docker system prune -af --filter "until=72h" on a timer.
• Upgrades: keep the runner within one minor version of GitLab; sudo apt install gitlab-runner upgrades it in place.
• Troubleshooting: sudo gitlab-runner verify, sudo journalctl -u gitlab-runner -f, and the job log header (which shows the image and executor).