# Lakehouse Plumber

> A modern, open-source, YAML-driven ETL framework for Databricks Lakeflow Spark Declarative Pipelines (SDP).

Lakehouse Plumber (LHP) turns reusable YAML configuration into readable native Python for Databricks pipelines. Load, transform, and write data, with test actions alongside the flow.

This is the Markdown representation of [the LHP homepage](https://lakehouse-plumber.dev/).

## One framework. Every part of your ETL.

- **Load from anywhere:** Auto Loader, Delta tables and Change Data Feed, SQL queries, JDBC databases, Kafka, Python functions, and custom PySpark data sources. See [load actions](https://mmodarre.github.io/Lakehouse_Plumber/reference/actions/load.html).
- **Transform in SQL or Python:** use SQL, Python, schema enforcement, data quality, and temporary tables. See [transform actions](https://mmodarre.github.io/Lakehouse_Plumber/reference/actions/transform.html).
- **Test your data:** check row counts, uniqueness, referential integrity, completeness, ranges, schemas, lookup coverage, and custom SQL or expectations. See [data tests](https://mmodarre.github.io/Lakehouse_Plumber/guides/test/data-tests.html).
- **Write to your destinations:** configure write actions and extend them through the documented interfaces. See [write actions](https://mmodarre.github.io/Lakehouse_Plumber/reference/actions/write.html).
- **Reuse patterns:** share templates, presets, and substitutions across pipelines. See [the guides](https://mmodarre.github.io/Lakehouse_Plumber/guides/index.html).

Read [the action model](https://mmodarre.github.io/Lakehouse_Plumber/concepts/the-action-model.html) to understand how actions connect through named views. Read the current documentation before using specific configuration fields or commands.

## Start with your agent

Give your coding agent the [setup guide](https://lakehouse-plumber.dev/agent.md). It covers an isolated Python environment, choosing a sample or empty project, and installing the LHP authoring skill with `lhp skill install`.

The [learning guide](https://lakehouse-plumber.dev/learn.md) provides temporary, docs-guided lessons. A dedicated teaching skill is not yet available. [Agent guides and prompts](https://lakehouse-plumber.dev/agents/) are also available as ordinary HTML.

## Install from your terminal

LHP requires Python 3.11 or later. In a Python virtual environment:

```sh
pip install lakehouse-plumber
```

Then follow the [Get Started course](https://mmodarre.github.io/Lakehouse_Plumber/get-started/index.html).

## Project resources

- [Documentation](https://mmodarre.github.io/Lakehouse_Plumber/)
- [Source code and releases](https://github.com/Mmodarre/Lakehouse_Plumber)
- [Python package](https://pypi.org/project/lakehouse-plumber/)
- [Issues](https://github.com/Mmodarre/Lakehouse_Plumber/issues)
- [Discussions](https://github.com/Mmodarre/Lakehouse_Plumber/discussions)
- [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0)
- [LHP telemetry documentation](https://mmodarre.github.io/Lakehouse_Plumber/reference/telemetry.html)
- [Website privacy and analytics choices](https://lakehouse-plumber.dev/privacy/)
- [Agent discovery index](https://lakehouse-plumber.dev/llms.txt)