MicroK8s 1.31: Three-Node HA, MetalLB, Ingress and Automatic TLS

A single MicroK8s node is a single point of failure. This post turns three machines into a highly available cluster, gives the ingress controller a stable address, and automates HTTPS certificates. It builds on the single-node install in the previous post and stays on the 1.31/stable track.

Architecture

flowchart TB
  USER[Users] -->|DNS app.example.com| VIP[MetalLB virtual IP<br/>10.0.1.200]
  VIP --> ING[Ingress NGINX<br/>on all three nodes]
  subgraph CL[MicroK8s HA cluster - dqlite quorum]
    N1[node1 control plane and worker]
    N2[node2 control plane and worker]
    N3[node3 control plane and worker]
  end
  ING --> N1
  ING --> N2
  ING --> N3
  N1 <-->|dqlite replication| N2
  N2 <-->|dqlite replication| N3
  N1 <--> N3
  CM[cert-manager] -->|ACME HTTP-01| LE[Lets Encrypt]
  CM -->|stores certificate| SEC[TLS Secret]
  SEC --> ING

MicroK8s HA uses dqlite, a Raft-based database: with three voters the cluster keeps working if one node fails (two of three form a quorum). Three nodes is the minimum for HA; two nodes give no fault tolerance for the datastore.

Form the cluster

On each of the three Ubuntu 24.04 machines install the same version, then join them:

sudo snap install microk8s --classic --channel=1.31/stable
# on node1: create a join token (single use)
microk8s add-node
# on node2 and node3: paste the printed command, for example
microk8s join 10.0.1.11:25000/<token>
# check
microk8s status | head -20          # shows "high-availability: yes" with 3+ nodes
microk8s kubectl get nodes

Generate a new add-node token for each member, and make sure the nodes can reach each other on ports 25000 (cluster agent), 19001 (dqlite), 16443 (API) and 10250 (kubelet). Use static IPs, synchronised clocks (chrony), and put the nodes in different failure domains. Node-to-node latency should be low; stretching dqlite over a slow WAN link causes leader elections.

Enable the add-ons on any one node (they apply cluster-wide):

microk8s enable dns rbac ingress metrics-server

MetalLB virtual IP

Bare-metal clusters have no cloud load balancer. MetalLB announces an IP from a pool you reserve (in layer 2 mode, one node answers ARP for the address and another takes over on failure).

microk8s enable metallb:10.0.1.200-10.0.1.210

The MicroK8s ingress add-on runs as a DaemonSet but does not create a LoadBalancer service, so add one:

apiVersion: v1
kind: Service
metadata:
  name: ingress
  namespace: ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 10.0.1.200
  selector:
    name: nginx-ingress-microk8s
  ports:
    - { name: http,  port: 80,  targetPort: 80 }
    - { name: https, port: 443, targetPort: 443 }

microk8s kubectl apply -f ingress-lb.yaml
microk8s kubectl -n ingress get svc ingress

Point your DNS A record (app.example.com) to 10.0.1.200 (or the public NAT address that forwards to it). Layer 2 mode sends all traffic for one IP through a single node at a time, which is fine for modest loads; for higher throughput use BGP mode with your router.

cert-manager and Let’s Encrypt

microk8s enable cert-manager

Create a ClusterIssuer. Start with the staging server while testing, to avoid Let’s Encrypt rate limits:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt-prod }
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef: { name: letsencrypt-prod-key }
    solvers:
      - http01:
          ingress: { class: public }

Now annotate the ingress and request TLS:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: demo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: public
  tls:
    - hosts: [app.example.com]
      secretName: app-example-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: web, port: { number: 80 } } }

microk8s kubectl -n demo get certificate
microk8s kubectl -n demo describe certificate app-example-tls
curl -I https://app.example.com

HTTP-01 validation needs port 80 reachable from the internet; for internal-only hosts or wildcard certificates use DNS-01 with your DNS provider’s API. cert-manager renews certificates automatically about 30 days before expiry.

Testing failure and operating the cluster

1. Power off node2: microk8s kubectl get nodes shows it NotReady, pods are rescheduled after the default eviction timeout, and the site stays up through the other nodes.
2. Power off the node holding the VIP: MetalLB moves the address within seconds. Verify with curl in a loop.
3. Replace a failed node: microk8s remove-node <name> (use --force if the node is gone), then add-node for the replacement.
4. Back up regularly: dqlite state lives under /var/snap/microk8s/current/var/kubernetes/backend; use microk8s kubectl exports for manifests and an application-level backup tool (for example Velero) for volumes.
5. Update nodes one at a time (snap refresh within the track), checking microk8s status between each.

Limits to remember

hostpath-storage is node-local, so a pod pinned to a failed node loses access to its volume until the node returns. Use replicated storage (for example the rook-ceph add-on or external NFS or cloud storage) for stateful workloads before calling this production.