On-Prem Platform: Databases, TLS and Zabbix

This continues the on-premises platform from the previous post: data services, internal TLS and monitoring on the three-node MicroK8s cluster.

Versions in use on this date. MicroK8s 1.32 ships its own Helm (microk8s helm3); Helm 3.17 was released in January 2025. Zabbix 7.0 LTS (released June 2024) is the current long-term-support line, and the Bitnami OCI charts on Docker Hub are the usual starting point for databases at this time.

Architecture

flowchart TB
  C[Clients on office network] -->|HTTPS| ING[Ingress<br/>certificate from internal CA]
  ING --> APP[apps namespace]
  APP -->|cluster-internal only| PG[PostgreSQL<br/>primary and replicas]
  APP --> MY[MySQL<br/>primary and replica]
  APP --> MG[MongoDB<br/>replica set of three]
  APP --> RD[Redis<br/>with Sentinel]
  CM[cert-manager<br/>CA ClusterIssuer] -.->|issues| ING
  PG --> BK[Scheduled dumps<br/>to NAS or object storage]
  MY --> BK
  MG --> BK
  ZB[Zabbix server and frontend<br/>separate VM with PostgreSQL] -.->|agent 2 and HTTP checks| PG
  ZB -.-> ING
  ZB -.->|Kubernetes API| API[API server]

Databases with Helm

Create the secrets first (in the data namespace from the previous post) so passwords never appear in a values file, then install each chart with replication enabled:

microk8s kubectl -n data create secret generic pg-credentials \
  --from-literal=postgres-password='<admin>' --from-literal=password='<app user>'

pg-values.yaml:

architecture: replication
auth:
  database: shop
  username: shop
  existingSecret: pg-credentials
primary:
  persistence: { size: 20Gi }
  resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { memory: 2Gi } }
readReplicas:
  replicaCount: 1
metrics: { enabled: true }

microk8s helm3 install pg oci://registry-1.docker.io/bitnamicharts/postgresql -n data -f pg-values.yaml
microk8s helm3 install redis oci://registry-1.docker.io/bitnamicharts/redis -n data \
  --set architecture=replication --set sentinel.enabled=true
microk8s helm3 install mongo oci://registry-1.docker.io/bitnamicharts/mongodb -n data \
  --set architecture=replicaset --set replicaCount=3
microk8s helm3 install mysql oci://registry-1.docker.io/bitnamicharts/mysql -n data \
  --set architecture=replication

Pin each chart version with --version, review the chart’s values before installing (the keys above are the common ones), and use helm get values to keep what you deployed in Git.

Operating rules:

• Expose databases only inside the cluster; a NetworkPolicy lets only the apps namespace connect (see the previous post).
• Place replicas on different nodes using pod anti-affinity (most charts expose an affinity or podAntiAffinity preset).
• Schedule dumps (pg_dump, mysqldump, mongodump) from a CronJob to storage outside the cluster, and test a restore each quarter. Replication protects against a node failing, not against an accidental DELETE.
• Running databases in Kubernetes is a trade-off. For heavily used stateful systems, consider a dedicated database VM or a Kubernetes operator that handles failover and backups.

Internal TLS with cert-manager

With no public DNS, create a private certificate authority, store it as a secret, and let cert-manager issue certificates from it:

openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -subj "/CN=Internal Platform CA" -out ca.crt
microk8s kubectl -n cert-manager create secret tls internal-ca-keypair --cert=ca.crt --key=ca.key

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: internal-ca }
spec:
  ca: { secretName: internal-ca-keypair }
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: apps
  annotations: { cert-manager.io/cluster-issuer: internal-ca }
spec:
  ingressClassName: public
  tls: [{ hosts: [shop.corp.example], secretName: shop-tls }]
  rules:
    - host: shop.corp.example
      http:
        paths: [{ path: /, pathType: Prefix, backend: { service: { name: shop, port: { number: 80 } } } }]

The MicroK8s ingress addon registers an ingress class named public. Distribute ca.crt to client devices (group policy or device management), keep ca.key offline after the cluster is set up where possible, and alert on certificate expiry. If you own a public domain, a DNS-01 challenge with a public CA avoids distributing a private CA at all.

Monitoring with Zabbix

Run Zabbix 7.0 LTS on a separate VM (server, web frontend and a PostgreSQL database), so monitoring stays up when the cluster has problems.

• Install Zabbix agent 2 on each node for operating-system metrics (CPU, memory, disk, network) and use its plugins for PostgreSQL, MySQL, MongoDB and Redis checks.
• For cluster state, use the Kubernetes templates (for example Kubernetes cluster state by HTTP and Kubernetes nodes by HTTP): create a read-only ServiceAccount, copy its token into the template macros, and point the API URL at the cluster. Zabbix documents running a proxy and agent inside the cluster with its Helm chart.
• Alert on: node not ready, pod restarts, persistent volume usage above 80 percent, database connection saturation, replication lag, backup job age, and certificates expiring within 21 days.

AWS emulation

Applications that expect AWS services can use LocalStack, deployed from its Helm chart into the cloud-emulation namespace. Treat it as a development and test convenience: it emulates AWS APIs but does not provide AWS durability or behaviour guarantees, so keep real data out of it.