Getting started

A pipeline is a YAML file with a source, optional transforms and a sink. Put it in pipelines/ and run dataflow apply.

Minimal pipeline

name: orders-to-warehouse
schedule: "*/15 * * * *"
source:
  postgres:
    dsn: ${PG_DSN}
    table: orders
    cursor: updated_at
sink:
  clickhouse:
    dsn: ${CH_DSN}
    table: raw.orders

Retries and alerts

Each batch retries up to five times with exponential backoff. After the last attempt the run is marked failed and an alert goes to the channel in alerts:.

Incremental sync

With cursor set, DataFlow stores the last seen value and reads only newer rows on the next run. Deleted rows are picked up from a soft-delete column if you name one in deleted_flag.

CLI

• dataflow apply — validate and schedule all pipelines
• dataflow run NAME — run one pipeline now
• dataflow logs NAME — show the last runs