data-pipeline topic

:sparkles: The present project is a basic process pipeline of extrating, transforming, loading, analysing and presenting. All of that was made by using suitable tools of web scraping, data analysis/pr...

GabrielMotaBLima

(project & tutorial) dag pipeline tests + ci/cd setup

marcosmarxm

A list of useful resources to learn Data Engineering from scratch

adilkhash

Making DAG construction easier

pipeline-tools

A list about Apache Kafka

infoslack

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

snowplow

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

msamogh

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

elementary-data

task management & automation tool

pydoit

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

kestra-io