Support ingest from NATs in sources

[gedw99]

Is there any interest / need for supporting NATS Jetstream ?

I suspect that supporting both might be quite easy since Pranadb is only subscribing to topic changes and not doing any publications to Kafka / nats

[purplefox]

Could you elaborate a bit on what you mean by supporting NATS Jetstream? Do you mean ingesting data from there into Prana?

Prana will be doing publishing to Kafka soon (sinks)

[gedw99]

Yes I mean that both Kafka and NATS Jetstream can be supported.

There is an abstraction library in golang that already does this . I just need to search for it on GitHub.

I guess the other aspect of this is if the team members want this. Nats used to kind of suck but it’s really stable and well featured these days

[purplefox]

Makes sense. So basically extend source to also work with NATS. Yes, I think that would be nice to have

[gedw99]

Hey @purplefox

Check out this: https://github.com/nats-io/nats-kafka/blob/main/docs/config.md

Its a Kafka / NATS bridge that works both ways. Nats --> Kafka and Kafka --> NATS.

It might be one one to achieve the Issue.. Or it might be over complicated. I am only playing with it for a day or two so far, and still undecided. So figured its worth raising here.

[gedw99]

sorry i have not had time to work through this.

i want to but got too many things.

if you have any ideas you want to discuss around this feel free to drag me in.

[gedw99]

@purplefox https://github.com/nats-io/k8s/tree/main/helm/charts/nats-kafka

its clearly been used for real. This makes running it quite easy.