The demo broker

One Redpanda in a container, and one Python file that fills it with eleven topics. A twelfth name is kept absent on purpose. The data is the same 2 400 fictional ACME partners the REST demo API serves. Every screen of the Kafka page reads it.

It ships with the repo. Two commands start and fill it:

docker compose -f docs/connectors/kafka/demo/docker-compose.yml up -d
python3 docs/connectors/kafka/demo/acme-topics.py

  ▸ created acme.partners
  ▸ created acme.orders
  ▸ created acme.partners-drift
  ▸ created acme.partners-poison
  ▸ created acme.partners-compacted
  ▸ created acme.partners-reserved
  ▸ created acme.partners-raw
  ▸ created acme.probes
  ▸ created acme.partners-trimmed
  ▸ created acme.empty
  ▸ created acme.live
  ✓ filled
  ✓ twelve topics as declared (acme.absent is absent)

The script needs the Kafka client for Python: python3 -m pip install confluent-kafka.

The topics

A partner has the same name and city here and on the REST demo. Each topic serves one section of the Kafka page:

Topic	Shape	Used by
`acme.partners`	2 400 JSON objects, one partition	Quickstart
`acme.orders`	24 000 orders, keyed by partner over six partitions	Partitions and the cursor
`acme.partners-drift`	2 400, and two new fields appear at message 1 201	Schema evolution
`acme.partners-poison`	1 000, of which 143 are not a JSON object	Poison messages
`acme.partners-compacted`	`cleanup.policy=compact`; 1 000 rows then 300 tombstones	Tombstones
`acme.partners-reserved`	500, each carrying a top-level `id` — the REST record verbatim	Seven reserved names
`acme.partners-raw`	500 with `Partner Name`, `credit.limit`, `MONTANT_€`, an object and an array	Names the producer chose
`acme.probes`	500 ChirpStack events: 450 `up` with the reading under `object`, 50 `status`; one reading is the text `sensor-fault`	A value nested in the message
`acme.partners-trimmed`	5 000, and `trim` moves its log start on demand	A log that moved under the cursor
`acme.empty`	created, zero messages	An empty poll
`acme.live`	empty until `feed` writes to it while a run reads	The head is the target
`acme.absent`	never created	A topic that does not exist

The poison messages (a truncated object, a JSON array, a JSON scalar) sit at fixed places, so every fill gives the same counts.

The three gestures that change the broker

These three write to the broker; the Kafka page says when to use each:

python3 docs/connectors/kafka/demo/acme-topics.py feed --rate 200 --seconds 30     # a producer racing the run, on acme.live
python3 docs/connectors/kafka/demo/acme-topics.py feed --topic acme.partners-trimmed --rate 500 --seconds 2
python3 docs/connectors/kafka/demo/acme-topics.py trim 5500                        # acme.partners-trimmed: offsets 0..5499 are gone

trim moves the partition's log start at once, as retention would over days. It runs rpk in the container lumnik-demo-redpanda. Set KAFKA_DEMO_CONTAINER to use another one.

Checking it before you blame the manifest

python3 docs/connectors/kafka/demo/acme-topics.py verify

It checks every topic's count and partitions, and that acme.absent does not exist. After trim, or feed --topic acme.partners-trimmed, verify reports acme.partners-trimmed: expected.

reset drops the topics and fills them again:

python3 docs/connectors/kafka/demo/acme-topics.py reset

Reset the hub's cursors too. With the old cursor, a run on a refilled topic reads nothing and completes. After reset, for each connector to read from the start:

lm endpoint reset-cursor <connector>/default

host.docker.internal:29092, not localhost:19092

Your shell (and acme-topics.py) reaches the broker at localhost:19092.
The hub, in its container, reaches it at host.docker.internal:29092 — what the manifests carry. The wrong listener ends in the broker timeout after about 60 s.

Check both:

docker exec lumnik-demo-redpanda rpk cluster info --brokers localhost:9092
docker compose -f docker-compose.selfhost.yml exec hub bash -c 'exec 3<>/dev/tcp/host.docker.internal/29092 && echo REACHABLE'

What it is not

Automatic topic creation is off, so acme.absent stays absent.
Authenticates nobody. Every topic is plaintext. The sasl-plain manifest on the Kafka page shows its refusals, not a successful handshake.
A throwaway rig. One node, no replication, no persistence to count on.