Connectors

Connectors ingest your legacy sources — files, databases, APIs, event topics — into hub tables, read-only and declaratively: one YAML manifest, three commands — validate, apply, run.

Which connector for which source?

Your source	Connector	Page
CSV exports — dropped locally, on SFTP, or in S3	`csv-file`	CSV & files
A live database — PostgreSQL; Oracle (Pro)	`jdbc-generic`	JDBC
An HTTP/JSON API (Stripe, HubSpot, …)	`rest-generic`	REST
A GraphQL API	`rest-generic`, GraphQL shape	GraphQL
A SOAP middleware (Siebel, SAP PI, …)	`rest-generic`, SOAP shape	SOAP
An event broker (Kafka, Redpanda, Event Hubs)	`kafka` — Pro	Kafka

There are five connector types: csv-file, jdbc-generic, rest-generic, kafka and hello-world (a smoke-test connector with no page of its own). GraphQL and SOAP are not types: they are request shapes of a rest-generic endpoint, with the same auth, retry and rate limits.

The pipeline

flowchart LR
  src["legacy source"] --> t["transport / engine"] --> p["parser"]
  p --> m["mapping & transforms"] --> tbl["hub table"] --> rag["RAG chunks"]

Every connector reads the source and writes a hub table. Each tenant sees only its own rows.

A csv-file connector can repair the data on the way: Transformers.
On lumnik Pro with the AI on, the table also feeds the scoped chat.

The lifecycle — every connector, same three commands

Assumes a deployed hub and an authenticated lm — see Quickstart.

Write a manifest (kind: Connector) — every connector uses the same envelope:

apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: <kebab-case-name>
  connector_type: <csv-file|jdbc-generic|rest-generic|kafka>
  scopes: [<scope>]                    # the scope(s) this connector's tables are askable in
spec:
  # type-specific — see your connector's page
  ...

lm validate -f manifest.yaml — check it without touching anything.
lm apply -f manifest.yaml — register or update the connector (safe to re-run).

lm connector run <name> — run it now. Or schedule it:

spec.schedule: "0 6 * * *" — cron, evaluated in UTC (here: daily at 06:00 UTC).
spec.window: "22:00-07:00" — scheduled runs start only inside this UTC window. The end is exclusive: 07:00 keeps the 06:00 slot inside.

What the three commands print, on a small orders export:

$ lm validate -f orders.yaml
✓ orders.yaml — valid
$ lm apply -f orders.yaml
Applied connector "orders" — 1 endpoint
$ lm connector run orders
run scheduled: 6806802f-ed3c-46d6-8eeb-a0f7cff94e77

"Scheduled" is not "done" — the result lands in the run ledger a few seconds later:

$ lm run list --connector orders
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
6806802f  orders     default   Completed  10  8    2        2             2026-10-19 01:41  2026-10-19 01:41
$ lm dlq list --run 6806802f-ed3c-46d6-8eeb-a0f7cff94e77
ID        CONNECTOR  RUN       REASON           ATTEMPTS  RESOLUTION  MESSAGE
7c863b7a  orders     6806802f  MalformedSource  1                     /tmp/exports/sales-orders.csv:10 — Record has 5 values but o…
004c3b56  orders     6806802f  MalformedSource  1                     /tmp/exports/sales-orders.csv:7 — Record has 4 values but on…

Ten rows read, eight written, two set aside in the dead-letter queue with their reason.

A connector with several endpoints starts one run per endpoint. lm connector run prints one run scheduled: line for each, and lm run list --connector <name> shows them all. 5. Watch it in the TUI (CLI & TUI): :sources (connectors), :runs (history), :dlq (rejected rows).

Several runs at once

A hub runs at most 4 scheduled endpoints at once. The others wait; a waiting endpoint has no run in lm run list yet.
lm run cancel <id> stops a run after its current batch of rows.
Raising the limit, and queuing manual runs: Turning on the run queue.

Freshness follows the schedule

A change in the source appears at the next run that reads it.

A row deleted at the source stays in the hub table. One exception: a csv-file connector reading one local file, where the table follows the file (full refresh).
A JDBC watermark connector re-reads only the rows whose watermark moved.

Validate before apply

Command	Purpose
`lm validate -f manifest.yaml`	Validate without applying. Exit 0 if valid (warnings allowed), 1 if errors.
`lm apply -f manifest.yaml`	Apply. The server runs the same checks: on errors nothing is written and you get the same report. Warnings do not block the apply.
`lm describe connector-type <kind>`	Show the schema of a connector type: fields, types, required/optional, docs, and the rules each key carries (see below).

What a connector type asks for

lm describe connector-type <kind> lists every key with its meaning. Under a key come the rules lm validate applies to it:

required when … — another key's value makes this one mandatory.
read only when … — under any other value, the key is ignored, and lm validate says so.
→ … — the hint a refusal gives.

At the end of a block come at least one of … and the keys that are refused when given. Excerpt, from lm describe connector-type csv-file:

$ lm describe connector-type csv-file
Connector type: csv-file

transport            REQUIRED  [NESTED, HIGH]
  Transport block
  kind                 REQUIRED  [STRING, HIGH]
    Transport mechanism
    one of: local, sftp, s3
  path                 optional  [STRING, HIGH]
    File path (local) or glob path (sftp)
    required when kind is local or sftp
    → Set path: /imports/*.csv (local) or a remote glob (sftp)
  host                 optional  [STRING, HIGH]
    SFTP host
    required when kind = sftp
  user                 optional  [STRING, HIGH]
    SFTP user
    required when kind = sftp
  port                 optional  [INT, LOW]
    SFTP port
  strict_host_key      optional  [BOOLEAN, LOW]
    Verify the SFTP host key against host_key_fingerprint
  host_key_fingerprint optional  [STRING, MEDIUM]
    SFTP host key fingerprint (SHA256:… as ssh-keygen -lf prints it)
    required when kind is sftp and strict_host_key is not off
    → Read it with `ssh-keyscan -p <port> <host> | ssh-keygen -lf -` (the run's refusal prints the same), confirm it out of band, then set host_key_fingerprint: "SHA256:…"
  …

Valid

$ lm validate -f stripe.yaml
✓ stripe.yaml — valid

Errors only

$ lm validate -f bad.yaml
✗ bad.yaml — 2 error(s), 0 warning(s)

  ERROR    spec.auth
          at least one of token_env, token_value is required when kind is bearer
          → Use `lm secret set <NAME> ...` then reference via token_env: <NAME>

  ERROR    spec.endpoints[0].id
          must match kebab-case identifier

Fix errors above before running 'lm apply -f bad.yaml'.
Error: validation failed: 2 error(s)

Warnings only (still valid, exit 0)

$ lm validate -f noscopes.yaml
! noscopes.yaml — 0 error(s), 1 warning(s)

  WARN     metadata.scopes
          no scopes declared — a user bound to a scope will not see this connector, its runs or its dead letters
          → Add: scopes: [compta]  (a user must hold the role scope:compta)

lm apply prints the same warnings, and applies. A warning never fails lm validate or lm apply.

Exit codes

Exit	Meaning
0	No errors (warnings allowed)
1	Validation errors, and every other failure (file not found, unparsable YAML, hub unreachable)