Operations

# Rolling restarts

Zero-lost-capacity deploys for the job container: health-gate the new supervisor, drain the old one.

## How a health-gated rolling restart works

Start new, prove healthy, then stop old.

Orchestrators with per-role health checks — Kamal distributions such as the [`dash` branch](https://github.com/mhenrixon/kamal), or any docker-level `HEALTHCHECK`-driven deploy — replace a job container in three steps: start the new container, poll its health check until it reports healthy, and only then `docker stop` the old one. If the new container never goes healthy, the old one keeps running.

Pgbus participates on both ends: the supervisor's standalone `/readyz` is the health gate for the *new* container, and the graceful-drain pipeline (`drain_timeout` / `shutdown_timeout`) bounds the stop of the *old* one.

## The readiness gate is container-local

The standalone /readyz answers for THIS supervisor, not the fleet.

When `health_port` is set, the supervisor's `/readyz` answers from its own state — never the database. That distinction matters precisely during a rolling deploy: a cluster-wide verdict would let a freshly-booted container pass the gate on the strength of the *old* container's still-heartbeating workers, and the orchestrator would stop the old container before the new one had forked a single child. (The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict — the two probes answer different questions.)

The body is a snapshot of the supervisor's fork table:

```json
{ "status": "OK", "expected": 3, "live": 3 }
```

| Status | HTTP | Meaning |
| --- | --- | --- |
| `BOOTING` | 503 | Connection not yet verified, queues not bootstrapped, or children not yet forked. |
| `OK` | 200 | Every child forked at boot is currently alive. |
| `DEGRADED` | 503 | A child died and is waiting out crash-restart backoff. |
| `DRAINING` | 503 | A stop signal arrived; the container is leaving. |

> **Tip:** A crash-looping replacement container never reaches `OK` — the deploy gate fails and the orchestrator keeps the old container running. That is the failure mode you want.

A clean worker recycle (`max_jobs_per_worker`, `max_memory_mb`, `max_worker_lifetime`) never flaps readiness: the snapshot refreshes after the reap-and-restart step of each monitor pass, so a recycled worker is already replaced by the time the next probe reads it.

## pgbus-health: the HEALTHCHECK probe

A dependency-free probe cheap enough for 1–5s intervals.

The gem ships a `pgbus-health` executable: plain Ruby and stdlib sockets, loading neither Bundler, nor Rails, nor the rest of the gem — so a docker `HEALTHCHECK` can run it every few seconds, and it works in images without curl. It GETs `127.0.0.1:<port>/readyz` and exits `0` on HTTP 200, `1` on anything else (non-200, refused, timeout), `2` on usage errors.

```shell
pgbus-health --port 9394                     # or PGBUS_HEALTH_PORT=9394 pgbus-health
pgbus-health --port 9394 --path /livez --timeout 2
```

Wire it into a Kamal role (`bundle binstubs pgbus` generates `bin/pgbus-health`):

```yaml
servers:
  job:
    hosts: [...]
    cmd: bin/pgbus start
    healthcheck:
      cmd: bin/pgbus-health --port 9394
      interval: 5s
      start_period: 30s   # cover Rails boot + queue bootstrap
    stop_timeout: 45      # must exceed pgbus shutdown_timeout
env:
  clear:
    PGBUS_HEALTH_PORT: 9394
```

## Aligning the shutdown budget

stop_timeout > shutdown_timeout > drain_timeout.

On `docker stop`, SIGTERM reaches the supervisor and readiness flips to `DRAINING`. Children stop claiming new work and finish in-flight jobs for up to `drain_timeout` (default 30s). The supervisor then waits `shutdown_timeout` — default `drain_timeout + 5` — before SIGKILLing stragglers. The orchestrator's stop grace period sits outside both:

```
orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
        45s                       35s (derived)                30s
```

> **Warning:** If the orchestrator's stop grace period is shorter than `shutdown_timeout`, docker SIGKILLs the whole tree mid-drain and the graceful path never finishes. Raising `drain_timeout` raises the derived `shutdown_timeout` automatically — raise the orchestrator's stop timeout to match.

An explicit `shutdown_timeout` below `drain_timeout` logs a boot warning: it guarantees mid-drain kills.

## The overlap window

Two supervisors briefly share the database — by design, safely.

Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires:

- Queue claims use `FOR UPDATE SKIP LOCKED` — a message goes to exactly one worker regardless of how many are reading.
- `single_active_consumer` queues arbitrate through session-level advisory locks, released by Postgres the instant a killed process's connection dies.
- Two live recurring schedulers dedup on the `(task_key, run_at)` unique record — the loser of the insert race skips the occurrence.
- Dispatcher maintenance is idempotent; two dispatchers just do some redundant work.

"One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.

## What a hard kill still costs

At-least-once holds, but read_ct counts deploy kills as failures.

Jobs killed past the drain window are redelivered after their visibility timeout — at-least-once holds. But PGMQ's `read_ct` increments exactly like a logical failure, so a long-running job that straddles *repeated* deploy kills can be pushed to the dead-letter queue without its code ever raising. `zombie_detection` logs exactly this pattern (`read_ct > 1` with no recorded failure for the message).

Keep jobs shorter than `drain_timeout`, or raise it (together with the orchestrator's stop timeout) for queues that can't be. For `idempotent!` event handlers there is a separate crash-window caveat tracked in [pgbus#385](https://github.com/zoolutions/pgbus/issues/385).