motojin.com, Inc

Data Pipeline

What is Data Pipeline?

A data pipeline is a process for moving data between source and target(destination, sink) systems.
Data pipelines consist of three essential elements:
 a source or sources
like RDBMS, CRMs, ERPs, Social, IoT
 a processing step or steps
like Transformation, Augmentation, Filtering, Grouping, Aggregation
like Data Lake, Data Warehouse

Data Pipeline vs ETL?

ETL stands for “extract, transform, load”.
ETL is a kind of data pipeline and refers to a specific type of data pipeline(usually just a sub-process). but Data Pipeline is the entire process involved in moving data from one location to another.