Jenkins: Agents, Credentials and Configuration as Code

A Jenkins controller configured by hand is a pet: nobody remembers exactly what was clicked, and a disk failure means days of rebuilding. Two practices fix that. Run builds on separate agents, so the controller only schedules, and describe the controller in Configuration as Code (JCasC) so a new one can be created from Git in minutes.

Versions in use on this date. Jenkins 2.387 LTS on Java 17, the JCasC plugin (configuration-as-code), the SSH Build Agents plugin and Docker 23.0 on the agent hosts.

Controller and agents

flowchart TB
  GIT[(Git repo with jenkins.yaml and plugins.txt)] --> CTRL
  subgraph CTRL[Jenkins controller - no builds, 0 executors]
    JCASC[JCasC loads jenkins.yaml at start]
    CREDS[(Credentials store<br/>encrypted)]
    SCHED[Scheduler]
  end
  SCHED -->|SSH, label docker| A1[Agent docker-1<br/>Java 17, Docker 23.0]
  SCHED -->|SSH, label docker| A2[Agent docker-2]
  SCHED -->|inbound WebSocket, label k8s| A3[Ephemeral pod agent]
  A1 --> B[Build and push image]
  CREDS -.->|injected into build, masked| B
  B --> REG[(Container registry)]

Add an agent

Prepare the agent host: Java 17, Git, Docker, a jenkins user and a work directory.

sudo apt install -y openjdk-17-jre git docker.io
sudo useradd -m -s /bin/bash jenkins && sudo usermod -aG docker jenkins
sudo -u jenkins ssh-keygen -t ed25519 -N '' -f /home/jenkins/.ssh/id_ed25519
# add the controller's public key to /home/jenkins/.ssh/authorized_keys

Anyone in the docker group is effectively root on that host. Use dedicated, disposable build agents for that reason and never share them with other workloads.

In Jenkins: Manage Jenkins, Nodes, New Node, type Permanent Agent, remote root directory /home/jenkins/agent, label docker, launch method Launch agents via SSH with a credential of type SSH Username with private key, and a Known hosts file or manually trusted host key verification strategy (do not pick “non verifying” in production). Inbound agents (using WebSocket mode behind your reverse proxy) suit agents in other networks, and ephemeral agents on Kubernetes or EC2 give every build a clean environment.

Use labels to match jobs to capabilities (docker, arm64, windows), and keep a few executors per agent according to CPU and memory.

Use credentials correctly

pipeline {
  agent { label 'docker' }
  options { timeout(time: 20, unit: 'MINUTES'); timestamps() }
  stages {
    stage('Build') {
      steps { sh 'docker build -t shop-api:${BUILD_NUMBER} .' }
    }
    stage('Push') {
      when { branch 'main' }
      steps {
        withCredentials([usernamePassword(credentialsId: 'registry-login', usernameVariable: 'REG_USER', passwordVariable: 'REG_PASS')]) {
          sh '''
            echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
            docker tag shop-api:${BUILD_NUMBER} registry.example.com/shop/api:${GIT_COMMIT}
            docker push registry.example.com/shop/api:${GIT_COMMIT}
          '''
        }
      }
    }
  }
  post { cleanup { cleanWs() } }
}

Use single-quoted Groovy strings with sh (as above) so the shell expands secrets rather than Groovy interpolating them into the log. Jenkins masks credential values in console output, but masking is only a safety net: do not print, base64-encode or write them to files that become artifacts. Create credentials under Manage Jenkins, Credentials with a folder scope when only one team’s jobs should use them, and prefer short-lived tokens (cloud OIDC or role assumption) over stored passwords wherever possible.

Configuration as Code

JCasC reads a YAML file at start-up and applies it. A starting jenkins.yaml:

jenkins:
  systemMessage: "Managed by JCasC - change via pull request"
  numExecutors: 0
  mode: EXCLUSIVE
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "admin"
          password: "${ADMIN_PASSWORD}"      # from an environment variable or secret file
  authorizationStrategy:
    globalMatrix:
      entries:
        - user: { name: "admin", permissions: [ "Overall/Administer" ] }
        - group: { name: "authenticated", permissions: [ "Overall/Read", "Job/Read" ] }
  nodes:
    - permanent:
        name: "docker-1"
        remoteFS: "/home/jenkins/agent"
        labelString: "docker"
        numExecutors: 2
        launcher:
          ssh:
            host: "10.0.2.21"
            credentialsId: "agent-ssh"
            port: 22
            sshHostKeyVerificationStrategy: "knownHostsFileKeyVerificationStrategy"
unclassified:
  location:
    url: "https://ci.example.com/"
credentials:
  system:
    domainCredentials:
      - credentials:
          - basicSSHUserPrivateKey:
              scope: SYSTEM
              id: "agent-ssh"
              username: "jenkins"
              privateKeySource:
                directEntry: { privateKey: "${AGENT_SSH_KEY}" }

Never put secrets in the file: use ${VAR} references filled from environment variables, Docker or Kubernetes secrets, or a secrets backend supported by the plugin. Exact keys differ between plugin versions, so use Manage Jenkins, Configuration as Code, Reference in your instance for the schema, and export current settings from the same page as a starting point.

Load it with the environment variable, then validate before restarting:

# in the systemd override or container env
CASC_JENKINS_CONFIG=/var/lib/jenkins/casc/jenkins.yaml

Manage Jenkins, Configuration as Code, Reload existing configuration applies changes without a full restart.

Pin plugins too

Configuration without the plugins it refers to does not rebuild. Keep a plugins.txt and install with the official tool:

configuration-as-code:<version>
workflow-aggregator:<version>
git:<version>
credentials-binding:<version>
ssh-slaves:<version>

jenkins-plugin-cli --plugin-file plugins.txt

For containers, build a custom image FROM jenkins/jenkins:lts-jdk17 (pin to an exact tag), run the plugin CLI in the Dockerfile, and copy jenkins.yaml in. Replace the <version> placeholders with the versions currently installed (the plugin manager lists them), and update them in pull requests.

Operating model

1. Change configuration by pull request to the JCasC repository; the controller reloads from Git on deploy.
2. Update plugins on a schedule (at least monthly), in a staging controller first, and apply security advisories promptly.
3. Back up JENKINS_HOME for job history and keys, but treat configuration as rebuildable.
4. Test a rebuild from scratch twice a year; if it takes more than an hour, find the manual step and automate it.