Health Checks That Lie: liveness, readiness and the Restart Loop

The most damaging line of YAML in many clusters is a liveness probe that checks too much.

It is written with good intentions: "if the app cannot reach the database, it is unhealthy, restart it." The result is that a slow database — a survivable degradation — becomes a restart storm that takes down the entire service.

The two probes answer different questions

• liveness: "is this process wedged beyond recovery?" If it fails, the process is killed and restarted. It must therefore only detect conditions a restart actually fixes.
• readiness: "can this instance take traffic right now?" If it fails, traffic is moved elsewhere. Nothing is killed.

A liveness probe decides whether to kill your process. It should be almost impossible to fail by accident.

A database being slow is not fixed by restarting your app. So a dependency check belongs in readiness (stop sending traffic to a struggling instance) and absolutely never in liveness (do not kill every instance at once).

The failure mode, step by step

1. The database gets slow.
2. Every replica's liveness probe starts timing out — all at once, because they share the dependency.
3. The orchestrator restarts every replica simultaneously.
4. Restarts add connection churn and cold caches, making the database slower.
5. Go to 1, with fewer replicas.

This is a cascade, and the probe wrote it.

A probe policy that holds up

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }   # process alive, event loop turning
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }   # deps reachable, cache warm, draining?
  periodSeconds: 5
  failureThreshold: 2

• /healthz checks only in-process health: can the event loop respond? No external calls.
• /readyz checks dependencies and warm-up state; it is allowed to fail often.
• Give liveness a generous failureThreshold so a single slow GC pause is not a death sentence.
• Add a startupProbe for slow-booting apps instead of a huge liveness grace period.

The Kubernetes probe documentation spells out the semantics; the discipline above is what stops the semantics from biting.

Health checks are a contract about what a restart fixes. Write them like one.