data-pipeline topic

:sparkles: The present project is a basic process pipeline of extrating, transforming, loading, analysing and presenting. All of that was made by using suitable tools of web scraping, data analysis/pr...

GabrielMotaBLima

(project & tutorial) dag pipeline tests + ci/cd setup

marcosmarxm

A list of useful resources to learn Data Engineering from scratch

adilkhash

Making DAG construction easier

pipeline-tools

A list about Apache Kafka

infoslack

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

snowplow

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

msamogh

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

elementary-data

task management & automation tool

pydoit

:zap: Universal Workflow Orchestration Platform — Code in any language, run anywhere. 800+ plugins for data, infrastructure, and AI automation.

kestra-io