✨ New PUBSUB - NATS JetStream support

[mfreeman451]

Pull Request Template

Description:

• Adding support for NATS JetStream message broker
• #584
• Highlight the motivation behind the changes and the expected benefits.

Breaking Changes (if applicable):

N/A

Additional Information:

• Using NATS JetStream API
• Include screenshots or code snippets (if necessary) to clarify the changes.

Checklist:

• [x] I have formatted my code using goimport and golangci-lint.
• [ ] All new code is covered by unit tests.
• [ ] This PR does not decrease the overall code coverage.
• [x] I have reviewed the code comments and documentation for clarity.

Thank you for your contribution!

[aryanmehrotra]

Hey, could you please check this out https://github.com/gofr-dev/gofr/blob/development/pkg/gofr/datasource/README.md

It is mentioned to have a separate repo - but you can also create a separate package, the same way it is done for mongodb, cassandra...

this should be supported as an external package - such that users who are using it will only have it in the binary.

[Umang01-hash]

@mfreeman451 also the tests are failing for various packages. Please fix it

[Umang01-hash]

@mfreeman451 Thank you for resolving the review comments. Now your PR contains failing tests and code. Please fix it.

[mfreeman451]

fixed the tests for some of the pubsub stuff failing, seeing some other stuff failing though, not sure if I am the cause of that? === RUN Test_SQLRetryConnectionInfoLog could not connect with 'user' user to 'test' database at 'host:3201', error: dial tcp 103.224.182.246:3201: connect: operation timed out sql_test.go:283: Error Trace: /Users/mfreeman/src/gofr/pkg/gofr/datasource/sql/sql_test.go:283 Error: "reading database configurations from config file\ngenerating database connection string for 'postgres'\nregistering sql dialect 'postgres' for traces\nconnecting to 'user' user to 'test' database at 'host:3201'\n" does not contain "retrying SQL database connection" Test: Test_SQLRetryConnectionInfoLog --- FAIL: Test_SQLRetryConnectionInfoLog (76.05s) FAIL FAIL gofr.dev/pkg/gofr/datasource/sql 76.538s

[Umang01-hash]

fixed the tests for some of the pubsub stuff failing, seeing some other stuff failing though, not sure if I am the cause of that? === RUN Test_SQLRetryConnectionInfoLog could not connect with 'user' user to 'test' database at 'host:3201', error: dial tcp 103.224.182.246:3201: connect: operation timed out sql_test.go:283: Error Trace: /Users/mfreeman/src/gofr/pkg/gofr/datasource/sql/sql_test.go:283 Error: "reading database configurations from config file\ngenerating database connection string for 'postgres'\nregistering sql dialect 'postgres' for traces\nconnecting to 'user' user to 'test' database at 'host:3201'\n" does not contain "retrying SQL database connection" Test: Test_SQLRetryConnectionInfoLog --- FAIL: Test_SQLRetryConnectionInfoLog (76.05s) FAIL FAIL gofr.dev/pkg/gofr/datasource/sql 76.538s

No @mfreeman451 you are not responsible for these test failures. They occur sometimes and can be corrected by re-running them

[vipul-rawat]

Hey @mfreeman451, I think the issue is with line 38 in go.mod of gofr, you are trying to import the package which has not been published!

Also, you are introducing the packages

• github.com/nats-io/nats.go v1.37.0 // indirect
• github.com/nats-io/nkeys v0.4.7 // indirect
• github.com/nats-io/nuid v1.0.1 // indirect But this is counterproductive as you've built the NATS datasource as a separate go module.

Not there are 2 things you could do -

1. Remove the go.mod from the NATS implementation which would resolve most of your problems(speculative)
2. Remove the pkg/gofr/datasource/nats dependency from the container and build them as 2 separate modules which just work as the interface defined by gofr is implemented by the nats implementation (refer to Mongo, Cassandra, etc.. implementation)

[mfreeman451]

Hey @mfreeman451, I think the issue is with line 38 in go.mod of gofr, you are trying to import the package which has not been published!

Also, you are introducing the packages

• github.com/nats-io/nats.go v1.37.0 // indirect
• github.com/nats-io/nkeys v0.4.7 // indirect
• github.com/nats-io/nuid v1.0.1 // indirect But this is counterproductive as you've built the NATS datasource as a separate go module.

Not there are 2 things you could do -

1. Remove the go.mod from the NATS implementation which would resolve most of your problems(speculative)
2. Remove the pkg/gofr/datasource/nats dependency from the container and build them as 2 separate modules which just work as the interface defined by gofr is implemented by the nats implementation (refer to Mongo, Cassandra, etc.. implementation)

The separate go.mod is a project requirement for new integrations, not mine. If you remove the nats dep from the container then you can't initialize the NATS pubsub..

[vipul-rawat]

Hey @mfreeman451, I think the issue is with line 38 in go.mod of gofr, you are trying to import the package which has not been published! Also, you are introducing the packages

• github.com/nats-io/nats.go v1.37.0 // indirect
• github.com/nats-io/nkeys v0.4.7 // indirect
• github.com/nats-io/nuid v1.0.1 // indirect But this is counterproductive as you've built the NATS datasource as a separate go module.

Not there are 2 things you could do -

1. Remove the go.mod from the NATS implementation which would resolve most of your problems(speculative)
2. Remove the pkg/gofr/datasource/nats dependency from the container and build them as 2 separate modules which just work as the interface defined by gofr is implemented by the nats implementation (refer to Mongo, Cassandra, etc.. implementation)

The separate go.mod is a project requirement for new integrations, not mine. If you remove the nats dep from the container then you can't initialize the NATS pubsub..

@mfreeman451 I understand how we need to declare new modules for the new integrations(such as they are not a direct dependency in the applications that are not using it). You can check #1054, container package to see how @aryanmehrotra implemented Azure EventHub as an external pubsub dependency.

[mfreeman451]

Anything else left to do here?

[Umang01-hash]

@mfreeman451 your PR is passing all the tests but failing the code quality step.

Here's what i found when i looked into it:

Please resolve these issues.

[aryanmehrotra]

@mfreeman451 any update on the issues?

[mfreeman451]

Removing context support from my integration is unacceptable, I am closing this Pull Request and will maintain my own fork of the integration and gofr that uses contexts on Connect() methods.