data-engineering topic

List data-engineering repositories

dagger

12
Stars
4
Forks
Watchers

Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).

airbyte

13.0k
Stars
3.4k
Forks
174
Watchers

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

awesome-billing

709
Stars
59
Forks
Watchers

💰 Billing & Payments knowledge for cloud platforms

awesome-dataops

102
Stars
9
Forks
Watchers

:sunglasses: A curated list of awesome DataOps tools

:sparkles: The present project is a basic process pipeline of extrating, transforming, loading, analysing and presenting. All of that was made by using suitable tools of web scraping, data analysis/pr...

airflow-testing-ci-workflow

81
Stars
10
Forks
Watchers

(project & tutorial) dag pipeline tests + ci/cd setup

Data-Engineering-HowTo

3.0k
Stars
429
Forks
Watchers

A list of useful resources to learn Data Engineering from scratch

dud

159
Stars
5
Forks
Watchers

A lightweight CLI tool for versioning data alongside source code and building data pipelines.

fastapi-dramatiq-data-ingestion

39
Stars
11
Forks
Watchers

Sample project showing reliable data ingestion application using FastAPI and dramatiq

Data-Engineering-Nanodegree

53
Stars
35
Forks
Watchers

Solution to all projects of Udacity's Data Engineering Nanodegree: Data Modeling with Postgres & Cassandra, Data Warehouse with Redshift, Data Lake with Spark and Data Pipeline with Airflow.