Skip to content

The WOW, step by step

Show, don't explain

A SOAP Siebel and a CSV become one canonical Customer — served over REST, then asked in natural language. Every file below is the real file the end-to-end test runs, so this walkthrough cannot drift away from what actually works.

What you will see

Connectors, entity fusion, sourced answers, and the first operational rungs above them.

Best reader

Someone who wants value before deep reference.

Then do this

Replay the same flow by hand on the deployed stack.

Why it matters

Sixty seconds of watching beats any feature list.

Read this one, run the other one

This page is the narrated WOW: the manifests and the commands are here so you can see what the flow is made of, but running them needs a hub, this repository on disk and a logged-in lm. To actually do it, follow The WOW, hands-on — same files, on the deployed self-host stack, with the prerequisites and the fake-siebel.py stub it needs.

Three parts of the flow are not in every build: the ask of §4 and the chat that answers §8's lifecycle questions both go through lumnik-rag, a module the open edition does not carry — nor lumnik-llm-langchain, where the model providers live. §5 has a foot in the same place: lm view generate needs a chat model; with lumnik-llm-langchain pruned it abstains with a named refusal ("no chat model is wired in this edition") rather than generating. §5's other half — writing the view YAML and rendering it with lm view -f — does not touch a model at all. Every other step here ingests, fuses and serves on that build. See which edition you installed.

Act 1 — ingest and fuse

1. A SOAP source

The first source is a fake Siebel SOAP connector. The manifest:

# Source 1 — the "legacy CRM": our fake Siebel over SOAP CustomerQueryPage.
# kind: soap is a request-shape on the HTTP engine (connector_type stays rest-generic).
# Lands canonical hub table connector.cust_siebel — the manifest fuses on its promoted columns.
apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: siebel-customers
  connector_type: rest-generic
  scopes: [demo]                  # the same métier as its CSV sibling: one bound user, both sources
spec:
  # fake-siebel.py runs on the HOST. From the hub container (self-host stack) reach it via
  # host.docker.internal; if you run the hub with `quarkus:dev` on the host, use localhost:8077.
  base_url: http://host.docker.internal:8077
  endpoints:
    - id: customers
      kind: soap
      path: /eai
      soap_action: CustomerQueryPage
      body: |
        <In><PageSize>{{page_size}}</PageSize><StartRowNum>{{cursor}}</StartRowNum></In>
      pagination:
        kind: soap
        page_size: 100
        last_page_path: "$.Body.Out.LastPage"
        response_path: "$.Body.Out.ListOfCustomer.Customer"
      incremental:
        mode: full
      target:
        kind: hybrid
        table: connector.cust_siebel
        unique_key: "$.CustCode"
        promote:
          - { name: customer_code, path: "$.CustCode", type: text }
          - { name: name,          path: "$.Name",     type: text }
          - { name: segment,       path: "$.Segment",  type: text }

lm apply -f docs/demo/wow-customer/siebel-connector.yaml
lm connector run siebel-customers   # applying declares it; this is what reads the source

2. A CSV source

The second source is a CSV export carrying the same customer universe from another angle.

# Source 2 — the "flat-file export": a CSV the second system spits out nightly.
#
# For csv-file connectors the hub table name is NOT configurable: it is
#   connector.t_<slug(metadata.name)>
# i.e. a hardcoded `t_` prefix + the slugified connector name. So:
#   metadata.name: cust-csv   ->   connector.t_cust_csv
# (that's the table the entity manifest fuses against). `target` is optional and carries
# only `indexes` — there is no table_prefix knob for csv-file.
apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: cust-csv                     # -> connector.t_cust_csv
  connector_type: csv-file
  scopes: [demo]
spec:
  transport:
    kind: local
    # Path AS SEEN BY THE HUB. On the self-host stack the demo folder is mounted at /demo
    # (docker-compose.wow.yml); with `quarkus:dev` on the host, use ./customer.csv.
    path: /demo/customer.csv
  parser:
    has_header: true                 # header row drives the column names (code, cname, email, ville)
  after_process: none                # leave the source file in place

Its data:

code,cname,email,ville
C-100,Jean Dupont,jd@x.fr,Paris
C-200,MARIE A.,m@x.fr,Lyon
C-300,Solo,solo@x.fr,Nice
C-500,Claire Martin,claire@x.fr,Lyon
C-600,Pierre Corneille,corneille@x.fr,Paris
C-700,Maurice Druon,druon@x.fr,Paris
C-800,Albert Camus,camus@x.fr,Lyon
C-900,Saint Exupery,exupery@x.fr,Lyon

lm apply -f docs/demo/wow-customer/csv-connector.yaml
lm connector run cust-csv
lm run list                         # both runs Completed before you fuse anything

Both CSV manifests on this page — this one and Act 3's — read /demo/…, a path inside the hub container. The demo overlay (docker-compose.wow.yml) is what mounts this folder there. Without it the run still reports Completed, with IN=0 OUT=0: the glob simply matched nothing.

3. Fuse into one canonical customer

The entity manifest declares the match and merge rules that unify both sources into one governed truth:

# The hero entity: one Customer fused from the two legacy sources on customer_code.
# Siebel is listed first, so on a disagreement Siebel is the representative and the CSV
# value is preserved in the _conflicts ledger (nothing is silently dropped).
kind: Entity
name: Customer
description: "A customer, fused from the Siebel CRM and the nightly CSV export."
match: [customer_code]
scopes: [clients]               # the métier(s) whose chat can ask about this entity
columns:                        # grounding fed to the chat + human labels a kind:View renders
  customer_code:
    label: "Code client"        # shown by kind:View instead of the raw canonical name
  name:
    label: "Nom"
  segment:
    label: "Segment"
    description: "Commercial segment."
    values: [GOLD, SILVER, BRONZE]   # the domain — a filter on any other value is REFUSED, not run
  city:
    label: "Ville"
    description: "Billing city (free text, French city names)."
    example: "Lyon"
sources:
  - table: cust_siebel          # source 1 (SOAP/Siebel) — promoted columns
    map:
      customer_code: customer_code
      name: name
      segment: segment
  - table: t_cust_csv           # source 2 (CSV export) — header columns
    map:
      code: customer_code
      cname: name
      email: email
      ville: city

lm entity apply -f docs/demo/wow-customer/customer-entity.yaml

4. Ask it

The first question proves the answer is computed over a fused truth that lived in neither source alone.

lm ask --scope clients "combien de clients à Lyon ?"
# → an answer computed over a fused truth that lived in neither Siebel nor the CSV alone,
#   with the SQL that ran shown beneath it.

Act 2 — the data starts working for you

(The doctrine behind this act — the chain, the channels, the named limits — is How lumnik breathes; this act runs it for real.)

See

View

Render the fused customer as a declarative, read-only list — reference.

React

Process

Emit a decision when a new Lyon customer lands — reference.

Decide

Inbox

Let a human acknowledge the event with an idempotent action — reference.

Remember

Workflow

Teach the substrate lifecycle and dormancy so silence becomes a signal — reference.

5. See it (kind: View)

# The first per-métier app surface: a list over the fused Customer entity (scope: clients).
# Read-only, card-backed — every column must exist in the clients SchemaCard or the render is refused.
apiVersion: apps.lumnik.io/v1
kind: View
metadata:
  name: clients-desk
spec:
  scope: clients
  table: connector.v_customer
  title: "Clients"
  columns: [customer_code, name, city, segment]

lm view -f docs/demo/wow-customer/clients-view.yaml --filter city=Lyon

Or skip writing the YAML — describe the view and let the AI write it, validated by the same gate (an unknown column can never slip through, and an unanswerable ask is refused with the reason):

lm view generate clients "les clients gold avec leur ville et segment" -o gold.yaml
cat gold.yaml            # the first line is the ready-to-run render command
lm view -f gold.yaml --filter 'segment=GOLD'

6. React to it (kind: Process)

When a new Lyon customer lands, emit a decision — delivery rides your existing webhook subscriptions (n8n here):

# The dragon's first rung, PROVEN LIVE: when a new customer from Lyon lands in the CSV,
# emit a decision event. The effect IS the event — delivery rides the existing webhook
# subscriptions (n8n/Make/Zapier), zero new code.
#
# Why `ville` and scope `demo` (not `segment`/`clients`): a process triggers on
# data.row.changed, born at the RAW write path — so on.table is the raw ingest table
# `connector.t_cust_csv`, and its columns are the CSV's own (code, cname, email, ville).
# `segment` is a Siebel field, absent from the CSV. The raw table lives in the `demo` scope
# card (the csv connector is tagged `demo`); the fused `clients` card names only v_customer.
apiVersion: apps.lumnik.io/v1
kind: Process
metadata:
  name: lyon-client-arrive
spec:
  scope: demo
  on:
    event: data.row.changed
    table: connector.t_cust_csv
  when:
    - ville=Lyon
  emit:
    type: decision.lyon-client-arrive
    message: "Nouveau client {cname} à {ville}"

lm process apply -f docs/demo/wow-customer/lyon-process.yaml
# or create it in the TUI: `:new` → Process — the same living example opens in $EDITOR

7. Decide (inbox → ack)

A process fires only on a genuinely new row. data.row.changed is born at the raw write path and dedups on the row's content hash, so re-running the same CSV emits nothing — and applying a process never fires retroactively over rows that already landed in step 2. Give it something new, then re-run the connector:

echo 'C-400,Camille Roux,camille@x.fr,Lyon' >> docs/demo/wow-customer/customer.csv
lm connector run cust-csv

customer.csv is code,cname,email,ville — pick a code the file does not already carry (C-400 is free) and a ville of Lyon, which is what lyon-process.yaml filters on. The process consumer ticks every 5s, so the decision lands a few seconds after the run.

Then the human disposes — and the ack is itself an event, idempotent by construction. The listing below is an illustration of the shape, not a transcript: the ID and the key are whatever your own run produced.

lm inbox
# ID    QUAND        PROCESS             MESSAGE                             ENTITY_KEY
# <id>  à l'instant  lyon-client-arrive  Nouveau client Camille Roux à Lyon  <row hash, 64 hex>
lm inbox ack <id>
# acked.

QUAND is a humanized age — under a minute it reads à l'instant. ENTITY_KEY here is the row's content hash, not a customer code: the event is emitted at the raw ingest table, before any entity key exists, and the inbox prints it whole — its 64 hex characters, unabridged. MESSAGE is lyon-process.yaml's "Nouveau client {cname} à {ville}" rendered from the row that fired it. Re-acking the same ID answers the same 200: the ack event is written once, ever.

8. Declare the lifecycle (kind: Workflow)

The three rungs above act on rows. This one teaches the substrate what the rows mean: the states a document moves through, the aliases twenty years of Excel produced for them, and which are terminal.

A lifecycle is declared over a column of real rows, so this act brings its own source — a commandes export whose statut column carries twenty years of drift:

# Act 3 source — the commandes export: the same nightly CSV habit as csv-connector.yaml,
# now landing the raw table the workflow act declares a lifecycle over.
#
# For csv-file connectors the hub table name is NOT configurable: it is
#   connector.t_<slug(metadata.name)>
# i.e. a hardcoded `t_` prefix + the slugified connector name. So:
#   metadata.name: commandes-csv   ->   connector.t_commandes_csv
# (that's the table commande-workflow.yaml's spec.on.table names). `target` is optional and
# carries only `indexes` — there is no table_prefix knob for csv-file.
apiVersion: connectors.lumnik.io/v1
kind: Connector
metadata:
  name: commandes-csv                # -> connector.t_commandes_csv
  connector_type: csv-file
  scopes: [demo]
spec:
  transport:
    kind: local
    # Path AS SEEN BY THE HUB. On the self-host stack the demo folder is mounted at /demo
    # (docker-compose.wow.yml); with `quarkus:dev` on the host, use ./commandes.csv.
    path: /demo/commandes.csv
  parser:
    has_header: true                 # header row drives the column names (num, client, statut, date_expedition)
  after_process: none                # leave the source file in place

lm apply -f docs/demo/wow-customer/commandes-connector.yaml
lm connector run commandes-csv       # lands connector.t_commandes_csv — seven commandes

Declare it only once those rows are in. Declaring first is allowed — the apply returns 200 over a table that does not exist yet — but then dataTruth has nothing to report and the whole point of the next paragraph disappears silently:

# Declare the document lifecycle of a column instead of teaching the chat its values by
# hand: states + labelled aliases (Excel reality is dirty: "en cours"/"encours" both mean
# the same thing) + a transition graph. provenance MUST be `declared` — v1 accepts no
# other value. `on.table`/`on.column` name the raw ingest column whose values this
# taxonomy explains; apply reports the data-truth confrontation: undeclared VALUES found
# in the data (reality, never refused — 'EN LITIGE' is one) and unobserved CODES declared
# but never seen.
apiVersion: apps.lumnik.io/v1
kind: Workflow
metadata:
  name: commande-lifecycle
spec:
  scope: demo
  provenance: declared
  # the sweep alerts a non-terminal commande observed unchanged past 30 days —
  # observation-relative, see docs/apps/workflows.md
  dormancy:
    after: 30d
  on:
    table: connector.t_commandes_csv
    column: statut
    key: num
  states:
    - code: BROUILLON
      initial: true
    - code: EN_COURS
      label: "En préparation"
      aliases: [en cours, encours]
    - code: TERMINE
      aliases: [fini]
      terminal: true
  transitions:
    - { from: [BROUILLON], to: EN_COURS }
    - { from: [EN_COURS], to: TERMINE }

lm workflow apply -f docs/demo/wow-customer/commande-workflow.yaml
# {"name":"commande-lifecycle","scope":"demo", … ,
#  "dataTruth":{"undeclared":[{"value":"EN LITIGE","rows":2}], … }}

Applying it doesn't hide the mess, it names it: EN LITIGE sits in the column and in no declared state, so the response reports it — and still returns 200, because reality doesn't ask the manifest's permission. The declared graph then lands in the scope's corpus, so the chat answers a lifecycle question from it instead of inventing one.

The manifest also declares spec.on.key (num) and spec.dormancy.after: 30d — a scheduled sweep watching every commande for the one thing the raw table itself cannot say: how long it has sat, unmoved, in a non-terminal state. Cold start counts from observation — nothing can be dormant before 30d elapses from lumnik's own first observation, never from the source's unknown history — and the alert never claims a source-absolute duration. To watch the sweep fire without the real 30-day wait, shorten the threshold in your own copy — spec.dormancy.after accepts minutes, so 1m is enough (the committed manifest still declares 30d). The listing below is an illustration of the shape, not a transcript: the ID is whatever your own run produced, and QUAND is the humanized age of §7 — à l'instant if you look right after the sweep. The … in MESSAGE is this page eliding for width, not the CLI abridging — lm inbox prints the alert in full, and Dormancy quotes the whole line:

lm inbox
# ID    QUAND        PROCESS                      MESSAGE                                                ENTITY_KEY
# <id>  à l'instant  workflow:commande-lifecycle  CMD-005 : observé immobile en « EN LITIGE » depuis …    CMD-005

CMD-002 and CMD-003 never alert — both resolve to the terminal TERMINE, which the sweep excludes. EN LITIGE — undeclared, and therefore never terminal — is watchable too: CMD-005 and CMD-007 sleep in it and alert exactly like any other stalled commande. See Dormancy for the full vocabulary.

After the demo — your data

The WOW you just read runs on the shipped dataset, and says so. The question every once-burned sponsor asks within a minute — "and on OUR data?" — has a road, and it is shorter than the demo suggests:

  1. Point at the real base, interactively. lm source jdbc test proves the connection, discover lists tables with row estimates and suggested watermark columns, add creates one connector per table with an encrypted credential — a SELECT-only account is all it ever needs (the guided flow).
  2. Expect the mess — it is the point. Fifteen years of mojibake, legacy dates and fantasy statuses don't block ingestion: transformers repair what should be repaired, and declaring a workflow gets you a data-truth report at apply time that names the dirty statuses it found — a report, never a police. The freezer's frost is inventory, not failure.
  3. Promote to a scope, ask the first question. Tag the connector with the scope, then run it (or lm source rediscover) — that is what writes the scope's schema card; from there the chat answers on your rows the same way it did on the demo's (what makes a scope answerable).

Boundaries: the mechanics above take hours, not days — what takes calendar time is what sits outside lumnik: the SELECT account, the network path, your DBA's diary. And two dependencies to have in place: the chat surfaces need lumnik Pro (which edition includes what), and the models must be pulled first.

Continue hands-on

Continue with the full by-hand replay — on the deployed self-host stack, real OIDC, lm login and Bearer-authenticated calls throughout (including the fake-siebel.py SOAP stub and the demo credentials).