Services and Cluster DNS

Pods come and go and their IPs change, so clients cannot depend on them. A Service gives a set of pods one stable virtual IP and DNS name, and load-balances across the pods that are Ready. Understanding the three layers (labels, endpoints, DNS) fixes most “my app cannot reach my other app” problems.

Versions in use on this date. Kubernetes 1.21, where the EndpointSlice API became stable (it replaces the single large Endpoints object for scalability), and CoreDNS 1.8 as the default cluster DNS.

From name to pod

flowchart TB
  C[Client pod] -->|1 resolve db.shop.svc.cluster.local| DNS[CoreDNS]
  DNS -->|ClusterIP 10.96.12.5| C
  C -->|2 connect to 10.96.12.5:5432| KP[kube-proxy rules<br/>iptables or IPVS on the node]
  KP -->|3 pick a Ready endpoint| P1[Pod db-0 10.244.1.7]
  KP --> P2[Pod db-1 10.244.2.9]
  SVC[Service selector app=db] -.-> ES[EndpointSlice<br/>lists Ready pod IPs]
  ES -.-> KP
  P1 -.->|labels app=db and readiness| ES
  P2 -.-> ES

The ClusterIP is virtual: no interface holds it. kube-proxy on every node rewrites traffic for it to a real pod IP.

Service types

Type	Reachable from	Typical use
ClusterIP (default)	Inside the cluster only	Internal APIs, databases
NodePort	<any node IP>:30000-32767	Simple tests, on-prem without load balancers
LoadBalancer	A cloud load balancer in front of NodePorts	Public services on EKS, GKE, AKS
ExternalName	Returns a CNAME to an external DNS name	Point an in-cluster name at an outside service
Headless (clusterIP: None)	DNS returns pod IPs directly	StatefulSets, client-side load balancing

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  type: ClusterIP
  selector: { app: web }
  ports:
    - name: http
      port: 80            # the Service's port
      targetPort: http    # the container port (named in the pod)

# a public service on AWS: an NLB or classic LB is created for you
apiVersion: v1
kind: Service
metadata:
  name: web-public
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
  type: LoadBalancer
  selector: { app: web }
  ports: [{ port: 443, targetPort: 8443 }]

For HTTP applications, prefer one Ingress in front of ClusterIP Services over many LoadBalancers, which each cost money (see the ingress post).

Cluster DNS names

Name	Resolves to
web	The Service in the same namespace
web.shop	Service web in namespace shop
web.shop.svc.cluster.local	Fully qualified name
db-0.db.shop.svc.cluster.local	A specific StatefulSet pod behind a headless Service
10-244-1-7.shop.pod.cluster.local	Pod IP form (rarely used)

Pods get search shop.svc.cluster.local svc.cluster.local cluster.local and ndots:5 in /etc/resolv.conf. That is why short names work, and also why external lookups such as api.github.com try several search suffixes first, adding latency. Use a trailing dot (api.github.com.) in hot paths or lower ndots in the pod’s dnsConfig.

Test and debug

kubectl -n shop get svc,endpointslices -l app=web      # does the Service have endpoints?
kubectl -n shop describe svc web                       # selector and Endpoints line
kubectl -n shop get pods -l app=web --show-labels       # do labels match the selector?

# DNS and connectivity from inside the cluster
kubectl run -it --rm dnscheck --image=busybox:1.33 --restart=Never -- nslookup web.shop.svc.cluster.local
kubectl run -it --rm curl --image=curlimages/curl:7.77.0 --restart=Never -- curl -sv http://web.shop/healthz

# from your laptop, bypassing Ingress
kubectl -n shop port-forward svc/web 8080:80

The checklist, in order:

1. No endpoints (empty Endpoints: line): the selector matches no Ready pods; fix labels or the failing readiness probe.
2. Name does not resolve: check CoreDNS pods (kubectl -n kube-system get pods -l k8s-app=kube-dns) and logs.
3. Resolves but times out: wrong targetPort, a NetworkPolicy blocking traffic, or the app listening on 127.0.0.1 instead of 0.0.0.0.
4. Works from some nodes: kube-proxy or CNI issue on that node.
5. LoadBalancer pending: no cloud controller or quota; kubectl describe svc events say why.

Good habits

• Name ports and refer to the names, so container port numbers can change without touching Services.
• One Service per application tier; do not point one Service at unrelated pods.
• Always define readiness probes so a starting pod receives no traffic.
• Use externalTrafficPolicy: Local on LoadBalancer Services when you need the client source IP preserved.
• Avoid fixed ClusterIPs and NodePorts unless required; let Kubernetes allocate them.