ETL (extract, transform, load)
Also called ELT, data pipeline.
A data pipeline that pulls data out of source systems, cleans and reshapes it, and loads it into a destination such as a data warehouse.
Last updated
What it is.
Extract reads data from apps, databases, files or APIs. Transform cleans it, joins it, fixes types and applies business rules. Load writes the result to the destination. In ELT, raw data is loaded first and transformed inside the data warehouse, which is common with modern warehouses.
Pipelines run on a schedule, such as nightly, or continuously from a stream of changes. Either way they need monitoring, because a silent failure means reports quietly go stale.
Why it matters
Most companies' data is spread across many tools. ETL is how it becomes one consistent set of numbers. It is also how data moves during system migrations, so its quality decides whether a migration goes smoothly.
In a product build
We make each pipeline step re-runnable without creating duplicates, which is idempotency applied to data. Every run logs row counts in and out, failed rows go to a review table, and alerts fire when a run fails or loads far fewer rows than usual.
Common mistakes.
- 01
Pipelines that fail silently, so dashboards show old numbers for days.
- 02
Transform rules buried in code that only one person understands.
- 03
Re-running a job and doubling the data.
Work it out with a tool.
Related terms.
- Data warehouseA database built for analysis and reporting, where cleaned data from many systems is kept together in one place.
- IdempotencyThe property of an operation that has the same effect whether it runs once or several times, so it is safe to retry.
- Message queueA buffer that holds messages, such as tasks or events, until a worker is ready to process them, so the sender does not have to wait.
- Event-driven architectureA way of designing software where parts of the system publish events about what happened, and other parts react to them on their own.
- Object storageA way of storing files as objects in buckets, each with an ID and metadata, reached over HTTP and built to scale to huge amounts of data cheaply.
