data-engineering topic
dagger
Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
awesome-billing
💰 Billing & Payments knowledge for cloud platforms
awesome-dataops
:sunglasses: A curated list of awesome DataOps tools
DS001--scraping-to-analysis--Extra-Store
:sparkles: The present project is a basic process pipeline of extrating, transforming, loading, analysing and presenting. All of that was made by using suitable tools of web scraping, data analysis/pr...
airflow-testing-ci-workflow
(project & tutorial) dag pipeline tests + ci/cd setup
Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
dud
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
fastapi-dramatiq-data-ingestion
Sample project showing reliable data ingestion application using FastAPI and dramatiq
Data-Engineering-Nanodegree
Solution to all projects of Udacity's Data Engineering Nanodegree: Data Modeling with Postgres & Cassandra, Data Warehouse with Redshift, Data Lake with Spark and Data Pipeline with Airflow.