On-call runbook — 2 a.m., by symptom
What is safe to restart, what each restart costs, and the exact commands — assembled once, instead of across three pages at 2 a.m.
Scope and access. This page is for the self-host compose stack. SSH to the host
that runs it; every command below runs from the repo root there (that is where
docker-compose.selfhost.yml and .env live). On Kubernetes, the costs column still
holds but the gestures are your cluster's (kubectl rollout restart …).
At 2 a.m., restart — never up
Re-running up.sh, or docker compose up -d with only the base file, re-composes
the stack without the TLS overlay — caddy vanishes and every device loses the
https origin (the overlay trap).
docker compose -f docker-compose.selfhost.yml restart <svc> touches one container
and cannot un-stack anything.
Step 0 — health before logs
curl -s http://localhost:8080/q/health
then the symptom → log routing tree. For the chat specifically,
know this before hunting for a dashboard: no chat/ask metric exists
(what /q/metrics does not carry) — the toolkit is /q/health, the
logs, and the ask-and-model symptom table.
"The chat is down" — the short path
/q/health— a DOWN names the sick service; no response at all during a boot means Flyway is still migrating (that is the signal — there is no named check)../deploy/selfhost/logs.sh hub— the log recipes have the jq one-liners.- The ask-and-model table: an opaque error after a
long wait = the model never loaded; nothing at all = Ollama unreachable (on Linux,
the bridge trap);
429= the hub's own rate limit.
The nightly login mystery, pre-solved
Logins failing for ~15 seconds at a fixed night hour: that is your backup cron briefly stopping Keycloak to copy its store (the docs' example schedules it at 03:10) — expected; already-issued tokens keep working throughout. Check your crontab's hour before opening an incident.
The restart matrix — cost and command
| Container | Safe? | Command (repo root) | What it costs |
|---|---|---|---|
hub |
yes | docker compose -f docker-compose.selfhost.yml restart hub |
In-memory rate-limit counters reset (harmless). A jdbc/CSV run in flight is interrupted and resumes on its cursor at the next trigger (how jdbc cursors resume); a REST run restarts its pagination from page one — no duplicates either way (content-hash / unique-key upserts). The compose healthcheck allows up to ~210 s before flagging unhealthy; a restart on an already-migrated DB is typically much faster. |
keycloak |
yes, briefly | docker compose -f docker-compose.selfhost.yml restart keycloak |
New logins fail during the window; already-issued tokens keep working. The store survives on its volume. |
postgres |
be gentle | docker compose -f docker-compose.selfhost.yml restart postgres |
A full outage while it lasts. The hub's pools recover, but expect a failed request or two right after (no connection validation is configured). Interrupted runs resume on their cursor afterwards. |
| Ollama (host) | yes | Linux: systemctl restart ollama · macOS/Windows: the Ollama app |
The model unloads: the first ask after restart is a cold load (tens of seconds). OLLAMA_KEEP_ALIVE=-1 keeps it resident (the trap list). |
caddy (façade) |
yes | docker compose -f docker-compose.selfhost.yml -f docker-compose.tls.yml restart caddy |
A blip on the HTTPS origin. Its state (ACME certs, the internal CA root your devices trust) survives on the caddy_data volume — which is exactly why the restart is a blip. |
After a hard kill: the zombie run reaps itself
A run killed mid-flight leaves its row at Running, and the scheduler skips any
connector that has one. The hub's reaper runs at startup and on every scheduler tick.
It fails a Running row only when no live process holds that endpoint's run lock, so
another replica's live run is never touched (error_message: reaped: …). It warns with the count and one line per run,
naming the run id and the connector whose schedule was blocked — so a crash costs at
most about a minute, never a silenced schedule: a run takes its lock before its row is
written, so the next scheduler tick (once a minute) reaps an abandoned row straight away.
The failed run's connector resumes on its cursor at the next trigger.
Read a reaped run's duration with that in mind. ended_at is when the reaper
marked the run dead, not when the run actually stopped. A run that died
at 02:00 while the hub was down until 09:00 shows seven hours in lm run list: that span is the
outage, not the ingestion. The run's own error_message says so, and lm run get
prints it.
What waits for morning
Anything touching the database's disk (see storage growth — nothing purges
the mirror), credential rotation, any lm connector delete (it purges data by design),
and any up-shaped gesture — see the red box above.