Load.
Transform.
Write.
Built for Databricks.
Lakehouse Plumber is a YAML-driven ETL framework for Databricks Lakeflow Spark Declarative Pipelines (SDP). Connect your sources, transform with SQL or Python, and write to the destinations you need—with data tests alongside the flow.
pip install lakehouse-plumber- name: load_orders_csv
type: load
source:
type: cloudfiles
path: "${landing_path}/orders/*.csv"
format: csv
target: v_orders_raw- name: clean_orders
type: transform
transform_type: sql
source: v_orders_raw
target: v_orders
sql: "SELECT order_id, customer_id, amount FROM stream(v_orders_raw)"- name: write_orders_bronze
type: write
source: v_orders
write_target:
type: streaming_table
catalog: "${catalog}"
schema: "${bronze_schema}"
table: ordersLoad from anywhere.
Bring files, tables, queries, databases, and event streams into the same action model. Use a built-in source or extend it with your own Python.
Explore load actions- Auto Loader
- Delta tables + Change Data Feed
- SQL queries
- JDBC databases
- Kafka
- Python functions
- Custom PySpark data sources
Simple in SQL. Flexible in Python.
Keep business logic in SQL or native Python. Add schema rules, data-quality checks, and temporary tables as the flow needs them.
Explore transform actionsSELECT
order_id,
customer_id,
amount
FROM stream(v_orders_raw)from pyspark.sql.functions import col
df.withColumn(
"order_id",
col("order_id").cast("bigint")
)Unit test your data.
Make row counts, uniqueness, relationships, and business rules part of the pipeline. Test actions check your data on pipeline updates and report the result.
Publish results through your own Python reporting provider.
Explore test actions- name: orders_id_unique
type: test
test_type: uniqueness
source: "${catalog}.${bronze_schema}.orders"
columns: [order_id]
on_violation: failEnable with --include-tests. Checks run during pipeline updates.
Write wherever your data needs to go.
Build native streaming tables and materialized views, publish to external sinks, or bring your own writer. The same write action carries your flow to its destination.
Explore write actions- Streaming tables (standard, CDC, snapshot CDC)
- Materialized views
- Delta sinks
- Kafka
- Azure Event Hubs (via Kafka)
- foreachBatch Python handlers
- Custom PySpark sinks
Build the pattern once. Reuse it everywhere.
LHP turns YAML into readable native Python for your pipelines. Shared templates and presets keep your patterns and defaults consistent. Update one shared definition across pipelines.
Your architecture. Your layers.
Use bronze, silver, gold, or any number of layers you need. Actions compose each flow; they don't prescribe your layers.
from pyspark import pipelines as dp
dp.create_streaming_table(
name="dev_catalog.bronze.orders",
comment="Streaming table: orders",
table_properties={"delta.enableChangeDataFeed": "true", "quality": "bronze"},
cluster_by=["order_id", "order_date"],
)
@dp.append_flow(
target="dev_catalog.bronze.orders",
name="f_orders_bronze",
comment="Append flow to dev_catalog.bronze.orders",
)
def f_orders_bronze():
df = spark.readStream.table("v_orders_raw")
return dfUpstream view v_orders_raw omitted for space.
Native Databricks. Code you can inspect.
Your pipelines use standard Lakeflow APIs, with no LHP runtime dependency. Open, inspect, and debug ordinary Python using your normal workflow. The code is yours.
Keep lasting changes in your action definitions, templates, and Python functions so they survive rebuilding the pipeline files.
Build with your agent, too.
Give your coding agent the LHP skill and your project conventions. It can compose load, transform, and write actions, add data tests, and reuse your patterns. You stay in control of the pipeline.
Genie Code
Claude
Codex
Gemini
Copilot“Load our orders, clean them with SQL, test order IDs, and write a streaming table using our shared patterns.”
From your first action to your next pipeline.
Start with installation, bring LHP into your coding-agent workflow, follow a task-focused guide, or look up precise configuration details.
Build your next pipeline with LHP.
pip install lakehouse-plumber