data-pipelines topic
dagger
Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).
dud
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
orchest
Build data pipelines, the easy way 🛠️
dagster
An orchestration platform for the development, production, and observation of data assets.
elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
optimus
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
deeplake
Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop....
versatile-data-kit
One framework to develop, deploy and operate data workflows with Python and SQL.
dataform
Dataform is a framework for managing SQL based data operations in BigQuery