pipeline topic
mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
nextflow
A DSL for data-driven computational pipelines
go_spider
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl com...
drake
An R-focused pipeline toolkit for reproducibility and high-performance computing
pytorch-toolbelt
PyTorch extensions for fast R&D prototyping and Kaggle farming
dawn
:sunrise: Dawn is a lightweight task management and build tool for front-end and nodejs.
datakit
Connect processes into powerful data pipelines with a simple git-like filesystem interface
go-streams
A lightweight stream processing library for Go
galaxy
Data intensive science for everyone.