Backup & Restore (self-host)¶

The self-host stack holds customer data in two stores:

Store	Contents	Backup method
PostgreSQL (`selfhost_pg` volume)	ALL hub data: ingested tables, RAG chunks, saved views, identities, audit	hot `pg_dump -Fc` — no downtime
Keycloak dev-file H2 (`selfhost_kc` volume)	realm changes made at runtime: users created by hand, password hashes	cold copy with a ~15s Keycloak stop

Why the brief Keycloak stop: the H2 file store is locked by the running server, and the admin API cannot export password hashes — a "live" export would silently lose credentials. New logins fail during the stop; already-issued tokens keep working.

Take a backup¶

deploy/selfhost/backup.sh              # keeps the newest 7
deploy/selfhost/backup.sh --keep 30
deploy/selfhost/backup.sh --no-keycloak

Output: backups/<UTC-stamp>/{lumnik.dump, keycloak-data.tgz} at the repo root (keycloak-data.tgz is intentionally absent when you passed --no-keycloak — its absence there is a choice, not a failed backup). The Keycloak capture excludes data/import — it's the read-only realm-import bind mount, reproducible from infra/keycloak/ in git, and restoring over it would fail.

Not covered — back these up separately, off-machine: - .env — contains LUMNIK_SECRET_MASTER_KEY; losing it makes secrets stored in the hub unrecoverable. Copy it somewhere safe once (it only changes when you rotate keys). - Host-level config (reverse proxy, cron entries).

Cron example¶

# nightly at 03:10, keep 14 days
10 3 * * *  cd /opt/lumnik && deploy/selfhost/backup.sh --keep 14 >> /var/log/lumnik-backup.log 2>&1

Ship backups/ to another machine (rsync, restic, object storage) — a backup on the same disk as the database is only half a backup.

Restore¶

Same machine (roll back data):

deploy/selfhost/restore.sh backups/<UTC-stamp>

The restore drops and recreates the lumnik database, then pg_restores the dump into the fresh copy — it does not restore in place with pg_restore --clean. That approach was tried first and found to be a dead end: core.audit_event is a declaratively partitioned table, and --clean emits a per-partition ALTER TABLE ONLY core.audit_event_default DROP CONSTRAINT audit_event_default_pkey, which PostgreSQL refuses because that primary key is inherited from the parent, not owned by the partition — the restore aborts with that error before the hub restarts. Drop-and-recreate sidesteps the inherited-PK limitation entirely and, as a bonus, also removes any object created after the snapshot was taken (an in-place --clean can only revert what's in the dump's TOC, so it can never remove something that didn't exist yet).

The Keycloak restore is an overwrite-merge: the snapshot's H2 store fully replaces the live one, but stale top-level entries absent from the snapshot may survive — harmless for this stack, where the store is exactly h2/ + transaction-logs/. The write-back streams a tar with uid/gid rewritten to the container's keycloak user (1000:0): a plain docker cp preserves host ownership, which leaves the H2 store read-only for Keycloak — it boots and serves reads, but every login/write fails with H2 error 90097 "The database is read only".

Fresh machine (disaster recovery):

Install docker, clone the repo, put your saved .env at the repo root.
deploy/selfhost/up.sh — init scripts create roles and an empty DB.
deploy/selfhost/restore.sh backups/<UTC-stamp>.

Known scar — issuer change orphans JIT identities¶

JIT-provisioned users are keyed (idp_issuer, idp_subject) in the hub DB. If the public origin (and therefore the Keycloak issuer URL) changed between backup and restore, restored identities point at the OLD issuer and logins re-provision fresh (orphaning history). Restore to the same public origin, or migrate identity.app_user.idp_issuer deliberately.

What this does NOT give you¶

Point-in-time recovery (it's a nightly snapshot, not WAL archiving).
Multi-node/k8s backup — the Helm path assumes your cluster's own volume snapshot tooling; only the PG dump portion translates directly.