jdbc-generic — Manifest reference¶

Declarative YAML manifest for the jdbc-generic connector type. One manifest = one JDBC datasource + N tables (each table becomes an endpoint). Apply via lm apply -f <file>. Validate before apply with lm validate -f <file>.

1. Envelope¶

apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: <kebab-case-name>     # required
  connector_type: jdbc-generic # required
  tags: [<tag>, ...]          # optional, merged with target.tags
spec:
  connection: { ... }
  defaults: { ... }           # optional, per-table override base
  source:
    tables: [{...}, ...]      # required, at least one
  schedule: "<cron>"          # optional
  target:
    tags: [<tag>, ...]        # recommended (warning if absent)

2. Connection block¶

connection:
  url: jdbc:postgresql://db.acme.io:5432/billing  # required
  user: ro_user                                    # required
  password_env: ACME_DB_PASSWORD                   # required — server env var (see below)
  dialect: postgres           # optional, inferred from url scheme; `postgresql` accepted as an alias

Supported dialects (v1): postgres. If dialect is declared and differs from the URL scheme, validation fails.

Password sources¶

At run time the password is resolved in priority order:

Attached credential — a Basic credential referenced by the connector (connector.credential_id), stored encrypted. The guided lm source jdbc add flow creates one automatically (named <source-name>--credential); manifests applied via lm apply don't set this.
password in config — plain-text fallback, dev/test only. Never put it in a manifest.
password_env — the name of a variable in the hub server's environment (OS env var, or a JVM system property). Unset at run time = a clear run failure. Note: unlike the CSV transports' key_env/password_env, this does not consult the lm secret registry — the value must exist in the server environment.

3. Source block — multi-table¶

Each entry in source.tables[] becomes a ConnectorEndpoint row at apply time. The endpoint name = table name; the endpoint path = /<table-name>.

source:
  tables:
    - name: invoices
      mode: watermark               # watermark | full
      watermark_column: updated_at  # required when mode=watermark
      pk_columns: [id]              # required, non-empty
      chunk_size: 1000              # optional, range 1..10000
      indexes:                      # optional — Postgres indexes on the hub table
        - { columns: [customer_id], unique: false }
    - name: customers
      mode: full
      pk_columns: [id]

Per-table `indexes`¶

Declarative indexes are per-table for jdbc (that is where the runtime reads them — spec.target.indexes is not read by this connector type). Each entry is { columns: [non-empty strings], unique?: bool } (unique defaults to false); malformed entries fail validation with a per-entry error instead of being silently dropped.

Watermark resume — composite cursor¶

The cursor is composite: {col: <watermark_column>, value: <last watermark>, pk: {<first pk column>: <last value>}}. Resume reads WHERE col > value OR (col = value AND pk > last_pk) — ties in the watermark column paginate stably via the first pk_columns entry as tiebreak. Scheduled runs are incremental for free: the persisted cursor carries the last watermark forward, so only rows changed since then are re-read. mode: full keeps a PK-only cursor (pk > last_pk), so full scans are chunked and resumable — it captures inserts by ascending PK but, having no watermark, never re-reads updated rows.

4. Defaults block¶

defaults provides per-table fallbacks. Each tables[i] field overrides the corresponding default. Shallow merge (no recursive merging).

defaults:
  chunk_size: 500
  schema: public
source:
  tables:
    - name: invoices
      mode: watermark
      watermark_column: updated_at
      pk_columns: [id]
      # chunk_size and schema inherited from defaults
    - name: line_items
      schema: billing               # overrides defaults.schema for this table only
      mode: watermark
      watermark_column: ts
      pk_columns: [invoice_id, line_no]
      chunk_size: 1000              # overrides defaults.chunk_size

5. Scheduling¶

schedule — optional cron expression for the connector as a whole. All tables ingest on the same trigger.

6. Target block¶

target.tags enables RAG scope filtering — strongly recommended. The connector warns at validate-time if missing.

Tables are created in the connector schema as connector.t_<name> (one per declared source table). A source column named id collides with the auto-generated id BIGSERIAL primary key — rename it in the column mapping before ingesting.

7. Guided flow — `lm source jdbc`¶

The manifest is not the only door: lm source jdbc walks the same ground interactively (the password is always read from stdin, never a flag):

printf '%s\n' "$DB_PASSWORD" | lm source jdbc test --url jdbc:postgresql://... --user ro_user
printf '%s\n' "$DB_PASSWORD" | lm source jdbc discover --url ... --user ... [--schema public] \
    [--skip-pattern 'pg_*,information_schema*,t_*'] [--include-views] [--output table|json]
printf '%s\n' "$DB_PASSWORD" | lm source jdbc add my-source --url ... --user ... \
    --tables invoices,customers [--template-mode llm-first|heuristic-only|dump-1-1] \
    [--schedule-cron '0 6 * * *'] [--interactive]
lm source jdbc list-tables my-source

test — connection check: dialect, server version, latency, visible schemas.
discover — lists tables with row estimates, columns, PK and a suggested watermark column.
add — creates one connector per table plus a mapping. --interactive opens a TUI table picker (pre-filled with the discovered watermark suggestions); otherwise pass --tables. The password is stored as an encrypted Basic credential (<source-name>--credential) attached to the connectors — it never lands in a config.
--template-mode picks how the mapping template is drafted: llm-first (default — the local LLM drafts it from sampled rows, falling back to the heuristic when no LLM is available or the draft is unusable), heuristic-only, or dump-1-1 (no transformation).

8. Full example¶

See lumnik-hub/src/test/resources/validator/jdbc-multi-table-valid.yaml for the canonical multi-table example used in tests.

9. Validation rules summary¶

Rule	Severity	Path
`connection.url` matches `jdbc:<dialect>:...`	ERROR	`spec.connection.url`
Dialect in supported set (postgres)	ERROR	`spec.connection.url`
Declared `dialect` matches URL scheme	ERROR	`spec.connection.dialect`
`source.tables[]` non-empty	ERROR	`spec.source.tables`
Each `tables[i].mode` in {watermark, full}	ERROR	`spec.source.tables[i].mode`
`mode=watermark` requires `watermark_column`	ERROR	`spec.source.tables[i].watermark_column`
`pk_columns` is a non-empty list	ERROR	`spec.source.tables[i].pk_columns`
`chunk_size` in [1, 10000] when present	ERROR	`spec.source.tables[i].chunk_size`
`indexes` entries are `{ columns: [non-empty strings], unique?: bool }`	ERROR	`spec.source.tables[i].indexes[j]`
No duplicate table names	ERROR	`spec.source.tables[i].name`
`schedule` (if present) is a valid cron	ERROR	`spec.schedule`
`target.tags` present	WARNING	`spec.target.tags`