Operations

Rolling restarts

Zero-lost-capacity deploys for the job container: health-gate the new supervisor, drain the old one.

How a health-gated rolling restart works#

Start new, prove healthy, then stop old.

Orchestrators with per-role health checks — Kamal distributions such as the dash branch, or any docker-level HEALTHCHECK-driven deploy — replace a job container in three steps: start the new container, poll its health check until it reports healthy, and only then docker stop the old one. If the new container never goes healthy, the old one keeps running.

Pgbus participates on both ends: the supervisor's standalone /readyz is the health gate for the new container, and the graceful-drain pipeline (drain_timeout / shutdown_timeout) bounds the stop of the old one.

The readiness gate is container-local#

The standalone /readyz answers for THIS supervisor, not the fleet.

When health_port is set, the supervisor's /readyz answers from its own state — never the database. That distinction matters precisely during a rolling deploy: a cluster-wide verdict would let a freshly-booted container pass the gate on the strength of the old container's still-heartbeating workers, and the orchestrator would stop the old container before the new one had forked a single child. (The Rails-mounted Pgbus::Web::HealthApp keeps the cluster-wide verdict — the two probes answer different questions.)

The body is a snapshot of the supervisor's fork table:

GET /readyz

{ "status": "OK", "expected": 3, "live": 3 }

Status	HTTP	Meaning
`BOOTING`	503	Connection not yet verified, queues not bootstrapped, or children not yet forked.
`OK`	200	Every child forked at boot is currently alive.
`DEGRADED`	503	A child died and is waiting out crash-restart backoff.
`DRAINING`	503	A stop signal arrived; the container is leaving.

A crash-looping replacement container never reaches OK — the deploy gate fails and the orchestrator keeps the old container running. That is the failure mode you want.

A clean worker recycle (max_jobs_per_worker, max_memory_mb, max_worker_lifetime) never flaps readiness: the snapshot refreshes after the reap-and-restart step of each monitor pass, so a recycled worker is already replaced by the time the next probe reads it.

pgbus-health: the HEALTHCHECK probe#

A dependency-free probe cheap enough for 1–5s intervals.

The gem ships a pgbus-health executable: plain Ruby and stdlib sockets, loading neither Bundler, nor Rails, nor the rest of the gem — so a docker HEALTHCHECK can run it every few seconds, and it works in images without curl. It GETs 127.0.0.1:<port>/readyz and exits 0 on HTTP 200, 1 on anything else (non-200, refused, timeout), 2 on usage errors.

shell

pgbus-health --port 9394                     # or PGBUS_HEALTH_PORT=9394 pgbus-health
pgbus-health --port 9394 --path /livez --timeout 2

Wire it into a Kamal role (bundle binstubs pgbus generates bin/pgbus-health):

config/deploy.yml

servers:
  job:
    hosts: [...]
    cmd: bin/pgbus start
    healthcheck:
      cmd: bin/pgbus-health --port 9394
      interval: 5s
      start_period: 30s   # cover Rails boot + queue bootstrap
    stop_timeout: 45      # must exceed pgbus shutdown_timeout
env:
  clear:
    PGBUS_HEALTH_PORT: 9394

Aligning the shutdown budget#

stop_timeout > shutdown_timeout > drain_timeout.

On docker stop, SIGTERM reaches the supervisor and readiness flips to DRAINING. Children stop claiming new work and finish in-flight jobs for up to drain_timeout (default 30s). The supervisor then waits shutdown_timeout — default drain_timeout + 5 — before SIGKILLing stragglers. The orchestrator's stop grace period sits outside both:

budget alignment

orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
        45s                       35s (derived)                30s

If the orchestrator's stop grace period is shorter than shutdown_timeout, docker SIGKILLs the whole tree mid-drain and the graceful path never finishes. Raising drain_timeout raises the derived shutdown_timeout automatically — raise the orchestrator's stop timeout to match.

An explicit shutdown_timeout below drain_timeout logs a boot warning: it guarantees mid-drain kills.

The overlap window#

Two supervisors briefly share the database — by design, safely.

Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires:

Queue claims use FOR UPDATE SKIP LOCKED — a message goes to exactly one worker regardless of how many are reading.
single_active_consumer queues arbitrate through session-level advisory locks, released by Postgres the instant a killed process's connection dies.
Two live recurring schedulers dedup on the (task_key, run_at) unique record — the loser of the insert race skips the occurrence.
Dispatcher maintenance is idempotent; two dispatchers just do some redundant work.

"One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.

What a hard kill still costs#

At-least-once holds, but read_ct counts deploy kills as failures.

Jobs killed past the drain window are redelivered after their visibility timeout — at-least-once holds. But PGMQ's read_ct increments exactly like a logical failure, so a long-running job that straddles repeated deploy kills can be pushed to the dead-letter queue without its code ever raising. zombie_detection logs exactly this pattern (read_ct > 1 with no recorded failure for the message).

Keep jobs shorter than drain_timeout, or raise it (together with the orchestrator's stop timeout) for queues that can't be. For idempotent! event handlers there is a separate crash-window caveat tracked in pgbus#385.