nats-server
nats-server copied to clipboard
nats-io
S3 Compatibility for NATS Object Store
Proposed change
Introduce a new module to NATS platform, which acts as a gateway for NATS object store. The new module, (let’s call it nats-s3-gateway, or nats-s3) will be a NATS client (JetStream client) but expose a REST API that adheres to the AWS S3 protocol. nats-s3 will be its own service, which abstracts away integration with NATS object store and provides S3 protocol implementation. See the PoC implementation here.
Use case
The S3 API is kind of a de facto standard for accessing Object Storage systems. Many object storage systems such as MinIO, SeaweedFS, JuiceFS, etc support S3 protocol. Having S3 protocol support NATS object store enables integration over HTTP.
Contribution
Yes, we are intending and interested in contributing to NATS. See the PoC implementation here.
We are currently working on a HTTP gateway for various aspects of NATS and for sure S3 compatible API would be a great addition.
Really great that you did a POC of this, we'll take a look when we get to thinking about the object store aspect of a above
Considering the scalability and bandwidth capabilities of S3, would it be possible to leverage S3 not only as an API but as a backend storage solution for NATS ?
This approach could offer two operational modes: 1) synchronously storing topics entirely on S3, and 2) using S3 as asynchronous cold storage for long-term or infinite topic retention, maintaining a local hot cache.
Adopting S3 as a primary backend could significantly enhance scalability compared to traditional storage and potentially render NATS nearly stateless (without local disk persistence), as seen with similar projects like Warpstream's adaptation of Kafka.
We are looking into many options, including that one. Most folks struggle with the latency, at least in the NATS.io ecosystem. They expect very fast response times both for ingress and egress from JetStream.
