timeplus-io
Add support for NATS Jetstream
Feature Request: Add Support for NATS Jetstream in Proton/Timeplus
Use Case
Proton/Timeplus is recognized for its high-performance streaming SQL capabilities, largely powered by ClickHouse. To enhance its integration capabilities and messaging support, it would be beneficial to add support for NATS Jetstream. NATS Jetstream offers advanced features like persistence, at-least-once delivery, consumer offsets, and replayability, which are essential for reliable and scalable message streaming in complex, distributed systems. By integrating NATS Jetstream, Proton/Timeplus can be leveraged in more diverse environments, particularly where NATS is already an integral part of the messaging infrastructure.
Desired Solution
The ideal solution would be the implementation of a NATS Jetstream connector within Proton/Timeplus. This connector should be capable of both publishing to and subscribing from NATS Jetstream topics. Key features would include:
- Stream Ingestion: Ability to ingest data streams from NATS Jetstream, enabling real-time analytics on the streaming data.
- Fault Tolerance: Ensuring at-least-once delivery semantics to guarantee message processing in the face of network or system failures.
- Scalability: Support for NATS Jetstream scalability features, including partitioning and load balancing across multiple consumers.
- Ease of Configuration: Simple configuration setup for connecting Proton/Timeplus with a NATS Jetstream server, including stream and consumer configuration.
- Data Transformation and Enrichment: Capability to perform SQL-based transformations and enrichments on the ingested data from NATS Jetstream.
Alternatives Considered
Currently, integrating NATS Jetstream with Proton/Timeplus requires custom solutions or middleware, which can be complex and less efficient. Other messaging systems like Apache Kafka are supported, but they do not align with the infrastructure where NATS Jetstream is already in use.
Additional Context
The addition of NATS Jetstream support would significantly enhance Proton/Timeplus's appeal in environments where lightweight and high-performance messaging is crucial. It would also align Proton/Timeplus with modern cloud-native architectures that are increasingly adopting NATS Jetstream for messaging.
Thank you @mfreeman451 to add this feature request. It's certainly something we badly want to add to Proton. Just checking whether you may want to contribute code on this.
No problem at all. Maybe we can invite external contributors to work on this and you can help us to review/test the integration. Let me setup a bounty on this. Our team right now are adding native ClickHouse support and will add more enhancements to the Kafka External Stream. If any one who are familiar with C++ and NATS to pick up this task, that'll be great
š $300 bounty ā¢ Timeplus
š $700 bounty ā¢ derekcollison
Steps to solve:
- Start working: Comment
/attempt #535with your implementation plan - Submit work: Create a pull request including
/claim #535in the PR body to claim the bounty - Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts
Thank you for contributing to timeplus-io/proton!
Add a bounty ā¢ Share on socials
| Attempt | Started (GMT+0) | Solution |
|---|---|---|
| š“ @oliverqx | Jan 31, 2024, 2:46:10 AM | WIP |
| š¢ @Sambit003 | Apr 2, 2024, 2:02:39 PM | WIP |
| š¢ @ayewo | May 20, 2024, 2:49:41 PM | WIP |
| š¢ @vishwamartur | Nov 19, 2024, 3:04:59 AM | WIP |
![algora-pbc[bot] avatar](https://avatars.githubusercontent.com/in/275896?v=4 "Published by a pub.dev verified publisher"){kind=small}
/attempt #535
| Algora profile | Completed bounties | Tech | Active attempts | Options |
|---|---|---|---|---|
| @oliverqx | TypeScript, Python | Cancel attempt |
Has not clickhouse already a NATS integration ?
no, clickhouse still does not fully support NATS Jetstream I guess? I find this PR still working in progress: https://github.com/ClickHouse/ClickHouse/pull/51281 full discussion here: https://github.com/ClickHouse/ClickHouse/issues/39459
We certainly can leverage some code of NATS integration in ClickHouse. Most likely it'll be a special Table Engine in ClickHouse. However in Proton, we prefer reading/writing NATS with External Stream, just like how it reads/writes Kafka, so that we can process latest events in NATS continuously, in the streaming fashion, instead of running a bounded query for the Table Engine multiple times to get the updates.
Has not clickhouse already a NATS integration ?
no, clickhouse still does not fully support NATS Jetstream I guess? I find this PR still working in progress: ClickHouse/ClickHouse#51281 full discussion here: ClickHouse/ClickHouse#39459
Hi,
OK; I see, Clickhouse has Core NATS but not acting as a Jetstream (durable) consumer.
Hi @Sambit003 , glad to see you pick up this task. Feel free to ask any questions here or in https://timeplus.com/slack
@jovezhong why this is?? No support for GNU Compiler Collections??
yes . we do not support gcc, plz use clang16. https://github.com/timeplus-io/proton/blob/develop/BUILD.md
historical reason, and we use forked llvm repo. personally, llvm is more friendly for learning.
Iād like to attempt this feature but probably wont be able to do anything substantive until next week or so.
/attempt #535
| Algora profile | Completed bounties | Tech | Active attempts | Options |
|---|---|---|---|---|
| @ayewo | Ā Ā Ā 3 timeplus bounties + 17 bounties from 4 projects |
TypeScript, Rust, Java & more |
Cancel attempt |
@ayewo hey, I'm working on this. So if I'm dropping it, I'll let you know. You can continue with my PR then.
Hi @deependujha , there were some GitHub users trying to work on this but I don't see PR for them since May. I will cancel those attempts and you can start working on them. It'll be C++ based to call NATS SDK to integrate with Timeplus Proton
