GitOps with Argo CD

In GitOps, Git is the single source of truth for what runs in a cluster. A controller inside the cluster continuously compares Git with reality and reconciles the difference. Deployments become merge requests; rollbacks become reverts; and the audit trail is the Git history. Argo CD is a widely used controller for this.

Versions in use on this date. Argo CD 2.2 (November 2021) on Kubernetes 1.23. Argo CD 2.3, which bundles the ApplicationSet controller, is due in the coming weeks; until then ApplicationSet is a separate install.

The pull-based model

flowchart TB
  DEV[Developer] -->|pull request| GIT[Git repository<br/>manifests Helm values Kustomize]
  CI[CI pipeline] -->|build and push image| REG[Container registry]
  CI -->|open PR bumping image tag| GIT
  GIT -->|merge to main| ARGO
  subgraph CLUSTER[Kubernetes cluster]
    ARGO[Argo CD<br/>repo-server application-controller] -->|renders and compares| LIVE[Live objects]
    ARGO -->|apply when OutOfSync| LIVE
    LIVE --> POD[Pods pull image from registry]
  end
  ARGO --> UI[UI and CLI<br/>health sync status diff]
  DRIFT[Someone runs kubectl edit] -.->|detected as drift<br/>self-heal reverts| ARGO

Because the cluster pulls from Git, CI does not need cluster credentials, which is a security gain over pipelines that run kubectl apply with admin kubeconfigs.

Repository layout

platform-config/
  apps/                 # Argo CD Application manifests (app of apps)
    shop-prod.yaml
    shop-staging.yaml
  shop/
    base/               # Deployment Service Ingress
    overlays/
      staging/kustomization.yaml
      prod/kustomization.yaml     # replicas, resources, image tag

Keep application source code and deployment config in separate repositories (or at least separate paths) so config changes do not trigger builds, and permissions differ.

Install

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.2.2/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d
kubectl -n argocd port-forward svc/argocd-server 8080:443
argocd login localhost:8080 --username admin --insecure

Change the admin password, then configure SSO (OIDC or SAML) and disable the local admin account for real use. Expose the UI through an Ingress with TLS and restrict access.

An Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-prod
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: shop
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: shop/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true          # delete objects removed from Git
      selfHeal: true       # revert manual changes
    syncOptions: [CreateNamespace=true, PruneLast=true]
    retry: { limit: 3, backoff: { duration: 10s, factor: 2 } }

argocd app list
argocd app get shop-prod
argocd app diff shop-prod
argocd app sync shop-prod
argocd app history shop-prod
argocd app rollback shop-prod 12      # temporary; prefer git revert (automated sync will re-apply Git)

Sync states: Synced / OutOfSync; health states: Healthy / Progressing / Degraded / Missing. Argo evaluates built-in health checks (Deployments ready, Ingress has an address) and you can add custom ones.

Projects and access control

AppProject limits which repos, clusters and namespaces an Application may use, so a team cannot deploy outside its area:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: shop, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/acme/platform-config.git"]
  destinations: [{ server: https://kubernetes.default.svc, namespace: shop }]
  clusterResourceWhitelist: []          # no cluster-scoped objects
  roles:
    - name: deployers
      policies: ["p, proj:shop:deployers, applications, sync, shop/*, allow"]
      groups: ["acme:shop-team"]

Secrets

Never commit plain Secrets. Common choices: Sealed Secrets (encrypted in Git, decrypted by a cluster controller), External Secrets Operator (syncs from AWS Secrets Manager or Vault), or SOPS with a plugin. Argo CD renders what is in Git, so secrets must arrive in a form it can apply safely.

Release flow

1. CI builds the image shop/web:1.7.3, runs tests and scans, pushes it.
2. CI (or Argo CD Image Updater) opens a pull request changing the tag in overlays/prod/kustomization.yaml.
3. Reviewers approve; merge to main.
4. Argo CD syncs; the Deployment performs its rolling update; probes decide when it is done.
5. Failure? git revert the commit. Argo applies the previous state. Avoid argocd app rollback as the permanent fix because Git would no longer match.

Promote across environments with pull requests that copy the tag from staging to prod, so production only receives versions that passed staging.

Operating notes

• Argo CD should manage itself and its Applications declaratively (app of apps) so the platform is reproducible.
• Ignore fields that controllers change legitimately (ignoreDifferences, for example replicas managed by an HPA).
• Use sync waves and hooks (argocd.argoproj.io/sync-wave) for ordering database migrations before application rollouts.
• Alert on Applications that are Degraded or OutOfSync for more than a few minutes (Argo exports Prometheus metrics).
• Keep the Argo CD version close to current; read the upgrade guide for each minor release.