rest-generic connector¶

Universal REST API source connector. Ingest from any HTTP/JSON API (Stripe, HubSpot, GitHub, Pennylane, …) into hybrid JSONB + typed columns + RAG embeddings.

Installer demo path? See docs/quickstart/installer.md for the 60-second sequence (Stripe / CSV / combined).

Quickstart¶

# 1. Generate manifest from an OpenAPI spec
lm source rest from-openapi https://api.stripe.com/openapi.json \
    --name stripe --tag billing --output stripe.yaml

# 2. Review and adjust (auth, pagination, schedule — the cron lands on the scheduler since A10)
$EDITOR stripe.yaml

# 3. Apply
lm apply -f stripe.yaml

# 4. Run the connector
lm connector run stripe

YAML manifest¶

apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: stripe
  connector_type: rest-generic
  tags: [billing]
spec:
  base_url: https://api.stripe.com
  auth:
    kind: bearer
    token_env: STRIPE_API_KEY
  default_rate_limit: 100/s
  endpoints:
    - id: charges
      method: GET
      path: /v1/charges
      pagination:
        kind: cursor
        cursor_param: starting_after
        cursor_path: $.data[-1:].id
        has_more_path: $.has_more
        response_path: $.data
      incremental: { mode: full }
      target:
        kind: hybrid
        table: ext.stripe_charges
        unique_key: $.id
        promote:
          - { name: amount,   path: $.amount,   type: bigint }
          - { name: currency, path: $.currency, type: text }
          - { name: status,   path: $.status,   type: text }

Each endpoint declares id (kebab-case), method (one of GET | POST | PUT | PATCH | DELETE), path, a pagination block and a target block; incremental is optional (defaults to full).

Pagination strategies¶

Strategy	Use for	Required fields	Optional fields (default)
`cursor`	Stripe, OpenAI, most modern APIs	cursor_param, cursor_path, has_more_path, response_path	cursor_binding (query)
`page`	Spring HATEOAS, Laravel	page_param, response_path	size_param, size (50), start_page (1), total_pages_path, has_more_path, last_path, empty_check (true)
`offset`	Legacy REST	offset_param, limit_param, response_path	limit (100), total_path
`link-header`	GitHub, JSON:API	response_path	rel (next)
`relay`	GraphQL Connections	cursor_path, has_more_path, response_path; plus cursor_param (query/header binding) or cursor_var (variable binding)	cursor_binding (query)
`body-link`	Django REST, JSON:API, NestJS, HATEOAS	link_path, response_path	—
`custom`	Anything weird	response_path	Kotlin script (deferred to a future release)

page stops on whichever of total_pages_path / has_more_path / last_path is present — and, with empty_check: true (the default), on the first empty page.

Cursor binding¶

For cursor and relay, cursor_binding picks where the next-page cursor is injected:

Binding	Cursor goes into	Needs
`query` (default)	a query parameter	cursor_param
`header`	an HTTP header	cursor_param (the header name)
`variable`	the POST body's `variables.<name>`	cursor_var — `relay` on a graphql endpoint only

variable is refused outside relay on a graphql endpoint (the cursor rides a POST body that a plain GET never sends); conversely, graphql endpoints require exactly relay + variable.

Pure GraphQL APIs (POST a query, cursor in variables): see graphql-generic.md.

Pagination presets¶

Modern APIs follow well-known conventions — declare one preset instead of the full block. A preset expands to a real strategy on lm apply.

Preset	For	Expands to
`spring-page`	Spring Data `Page`	`page` (`$.content`, `$.totalPages`, 0-indexed)
`spring-slice`	Spring Data `Slice` (no total)	`page` + `last_path: $.last`
`spring-hateoas`	Spring Data REST	`body-link` (`$._links.next.href`, `$._embedded.{items}` — needs `items`)
`micronaut`	Micronaut Data `Page`	`page` (`$.content`, `$.totalPages`)
`drf-page`	Django REST `PageNumberPagination`	`body-link` (`$.next`, `$.results`)
`drf-limit-offset`	Django REST `LimitOffsetPagination`	`offset` (`$.count`, `$.results`)
`fastapi`	fastapi-pagination	`page` (`$.items`, `$.pages`, 1-indexed)
`nestjs`	nestjs-paginate	`body-link` (`$.links.next`, `$.data`)
`json-api`	JSON:API	`body-link` (`$.links.next`, `$.data`)
`sequelize`	Node/Sequelize	`offset` (`$.count`, `$.rows`)
`stripe`	Stripe-style cursor	`cursor` (`$.has_more`, `$.data`)

pagination: { kind: spring-page }                       # 7 lines → 1
pagination: { kind: spring-page, size: 50 }             # override any field
pagination: { kind: spring-hateoas, items: products }   # {items} → $._embedded.products

Adding a convention is one CDI bean (PaginationPreset) — the engine, validator, and apply handler are untouched.

body-link is also a first-class strategy: { kind: body-link, link_path: $.next, response_path: $.results } follows a next URL from the response body until it is null.

Auth providers¶

Kind	Config
`bearer`	`token_env: VAR_NAME` (preferred), or `token_value: "..."` (dev only)
`basic`	`username` + `password_env: VAR_NAME`
`api-key`	`in: header\\|query`, `name`, `value_env: VAR_NAME`
`none`	no credentials (the default when `auth:` is omitted)

Every *_env field resolves an lm secret name first, then falls back to a server environment variable / JVM system property of that name.

OAuth2 (client_credentials, authorization_code), HMAC, and mTLS are dedicated future projects.

Request headers¶

Connector-level only — spec.headers[] is applied to every request:

spec:
  headers:
    - { name: X-Api-Version, value: "2024-10" }       # static
    - { name: X-Client-Id,   value_env: CLIENT_ID }   # from the environment

Each entry needs name + exactly one of value / value_env.
value_env resolves the OS environment first, then a JVM system property.
Headers are applied before auth — auth-injected headers win on a name clash.
Endpoint-level headers are refused at apply ("per-endpoint headers are not supported in this release; declare headers at spec.headers").

Reliability¶

Retry — fixed policy, not a manifest knob: up to 5 attempts per request on HTTP 429, 500, 502, 503, 504, 408 and on network errors; exponential backoff 1s → 2s → 4s → … capped at 60s, ±10% jitter; a numeric Retry-After header overrides the computed delay. Any other status is returned as-is, no retry.
Page cap — the pagination loop stops after 10 000 pages per endpoint per run (MAX_PAGES), a runaway-loop guard.
Rate limit — default_rate_limit (default 100/s) is a count/window string; the window is an optional integer + unit s|m|h (100/s, 100/5m, 1000/h). Declared once at connector level: one token bucket per connector, shared by all its endpoints — no per-endpoint override.

Incremental sync¶

incremental:
  mode: full              # always re-fetch all pages (default)
  # OR
  mode: watermark         # send the stored watermark as a query param,
  param: updated_since    #   e.g. ?updated_since=<watermark>
  watermark_path: $.updated   # new watermark = MAX of this JsonPath over the items

mode: api_cursor (API-native cursor persistence) is deferred — not implemented; declaring it today behaves like full.

Cursor state is stored in connector.connector_cursor per endpoint and survives runs.

Storage layout¶

Each endpoint writes to a dynamically created ext.<connector>_<endpoint> table:

CREATE TABLE ext.stripe_charges (
    id            BIGSERIAL PRIMARY KEY,
    tenant_id     BIGINT NOT NULL,
    external_id   TEXT NOT NULL,
    -- promoted typed columns:
    amount        BIGINT,
    currency      TEXT,
    status        TEXT,
    -- always:
    data          JSONB NOT NULL,
    indexed_at    TIMESTAMPTZ DEFAULT now(),
    connector_id  UUID NOT NULL,
    endpoint_id   UUID NOT NULL,
    UNIQUE (tenant_id, external_id)
);
CREATE INDEX ON ext.stripe_charges USING GIN (data jsonb_path_ops);

UNIQUE (tenant_id, external_id) + ON CONFLICT DO UPDATE makes re-syncs idempotent.

Deleting a REST connector keeps its data

lm connector delete purges connector.t_* dynamic tables (CSV/JDBC) but does not drop REST ext.* hybrid tables — wipe an endpoint's rows explicitly with lm endpoint purge <connector>/<endpoint>.

CLI¶

lm apply -f <file.yaml>
lm endpoint enable|disable|reset-cursor|purge <connector>/<endpoint>
lm connector run <connector>
lm source rest test <connector>
lm source rest from-openapi <url> --name N --tag T -o FILE
lm source rest probe <connector> --endpoint <path>

TUI¶

:connectors — KIND column shows rest/jdbc/csv
Enter on a REST connector → :endpoints (drill-down)
:endpoints — NAME/METHOD/PATH/SCHEDULE/LAST_RUN/STATUS; Enter → :runs filtered

Troubleshooting¶

429 / rate limit ban: lower default_rate_limit (e.g., 100/s → 50/s)
Cursor stuck: lm endpoint reset-cursor <connector>/<endpoint> → next run is full refresh
Transforms: a promote entry takes an optional transform: (from_epoch, parse_iso8601, to_lower, to_upper, trim, to_json_string) applied at write time — an unknown transform is refused at apply. See Transform at ingestion.
Schema drift: edit YAML to add new promote columns; next run does ALTER TABLE ADD COLUMN
Wipe all data for an endpoint: lm endpoint purge <connector>/<endpoint> (requires confirmation)