[Extension Proposal] The Pipeline Framework

[mbarcia]

Description

The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, orchestrate and deploy pipeline steps with automatic gRPC and REST adapter generation.

Product Owners

• Microservices Architecture: Each pipeline step can be deployed and scaled independently
• Data Integrity: Immutability ensures that all transformations are preserved, providing robust data integrity
• Audit Trail: Every input transformation is permanently recorded, enabling complete auditability
• Reliability: Built-in dead letter queue for error handling ensures resilient processing

Architects

• Immutability by Design: No database updates during pipeline execution - only appends/preserves
• Kubernetes & Serverless Ready: Native builds enable quick start-up times for Kubernetes and serverless platforms (AWS Lambda, Google Cloud Run)
• Cost Efficiency: Reduced resource consumption with native builds and reactive processing
• Scalability: Each pipeline step can be scaled independently based on demand
• gRPC & REST Flexibility: High-performance gRPC for throughput-intensive tasks, REST for ease of integration
• Multiple Persistence Options: Choose from reactive (Panache) or virtual thread-based persistence models
• Rich Step Types: Support for OneToOne, OneToMany, ManyToOne, ManyToMany, and SideEffect processing patterns

Developers

• Monorepo Support: Complete pipeline can be debugged step-by-step in a single repository
• Compile-Time Safety: All entities in common package provide type safety across steps
• Reactive Processing: Mutiny's event/worker thread model for efficient concurrency
• Build-Time Generation: Automatic gRPC and REST adapter generation at build time
• gRPC & REST Flexibility: Fast gRPC for high-throughput scenarios, REST for ease of development and integration
• TLS Ready: Reference implementation includes TLS configuration out of the box
• Multiple Processing Patterns: OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect and blocking variants
• Health Monitoring: Built-in health check capabilities for infrastructure monitoring

QA Engineers

• Unit Testing Excellence: Each step deliberately kept small, making unit tests easy to write and fast to run
• Deterministic Behavior: Immutability ensures consistent test results across runs
• AI-Assisted Development: Modular structure is ideal for AI coding assistants
• Focused Testing: Each step's limited scope enables comprehensive testing
• Multiple Execution Modes: Test with different processing patterns (OneToOne, OneToMany, etc.) to validate behavior
• Health Check Testing: Built-in health monitoring provides additional testable endpoints

CTOs

• Cost Reduction: Native builds and reactive processing reduce infrastructure costs
• Developer Productivity: Rapid development with visual designer and auto-generation
• Operational Excellence: Built-in observability, health checks, and error handling
• Technology Modernization: Reactive, cloud-native architecture with industry standards
• Risk Mitigation: Dead letter queue and multiple processing patterns provide resilience

✨ Capabilities summary

• Reactive, immutable architecture with strong compile-time safety
• Automatic adapter generation for REST & gRPC
• Modular monorepo with clear separation of runtime vs. deployment
• Native-ready builds for ultra‑fast startup and cloud efficiency
• Built-in health monitoring, observability, metrics, and tracing
• Visual pipeline designer for rapid development
• Multiple execution patterns (OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect, blocking)
• Comprehensive error handling with dead-letter queue support
• Multiple persistence models (reactive or virtual‑thread‑based)
• Strong testing support: deterministic, focused, and easy to automate

Interested in this extension, please +1 via the emoji/reaction feature of GitHub (top right).

Repository name

quarkus-pipeline

Short description

The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.

Repository Homepage URL

https://pipelineframework.org

Repository Topics

• quarkus-extension
• grpc
• microservices
• rest
• reactive
• functional
• immutability
• pipeline
• monorepo
• dlq
• health-checks
• native
• observability

Team Members

• Mariano Barcia

Additional context

No response

[quarkus-bot[bot]]

/cc @aloubyansky (extension-proposal), @gastaldi (extension-proposal), @geoand (kubernetes), @gsmet (extension-proposal)