data-pipeline topic

:sparkles: The present project is a basic process pipeline of extrating, transforming, loading, analysing and presenting. All of that was made by using suitable tools of web scraping, data analysis/pr...

GabrielMotaBLima

(project & tutorial) dag pipeline tests + ci/cd setup

marcosmarxm

A list of useful resources to learn Data Engineering from scratch

adilkhash

Making DAG construction easier

chriscardillo

A list about Apache Kafka

infoslack

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

snowplow

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

msamogh

Open-source data observability for analytics engineers.

elementary-data

task management & automation tool

pydoit

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

kestra-io