New Opensearch sink

[arunx2]

Added new OpenSearch output component. I just opened a PR for initial review and planning to make some more changes for metrics and use public config API

[Jeffail]

Hey @arunx2, any chance you could shift this over to the new public plugin APIs? A good example would be something like the jetstream input: https://github.com/benthosdev/benthos/blob/main/internal/impl/nats/input_jetstream.go. Otherwise I can do the migration myself but might need to wait a little while.

[arunx2]

I have that in my to-do. I'll do it in a day or two

Thanks Arun

On Mon, Aug 22, 2022, 5:09 AM Ashley Jeffs @.***> wrote:

Hey @arunx2 https://github.com/arunx2, any chance you could shift this over to the new public plugin APIs? A good example would be something like the jetstream input: https://github.com/benthosdev/benthos/blob/main/internal/impl/nats/input_jetstream.go. Otherwise I can do the migration myself but might need to wait a little while.

— Reply to this email directly, view it on GitHub https://github.com/benthosdev/benthos/pull/1388#issuecomment-1222140103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGOCTYF52SBR3QMC7NIN5LV2NGXBANCNFSM56YCKAZQ . You are receiving this because you were mentioned.Message ID: @.***>

[arunx2]

I tried to migrate to new public API, and it looks like we might need to implement helper methods for handling basic authentication, optional AWS configuration like we did for TLS configuration and back off configurations.

[arunx2]

@Jeffail @mihaitodor Can you review when you get time? I've some cycles to work on this week and the next. Thanks!

[arunx2]

Hey @arunx2, thanks for this contribution! Looks good overall. I haven't tested the code yet, but I did do an initial read through it and left some comments

I updated the code. Appreciate if you can take a look again.

[arunx2]

@Jeffail I'm happy to work on next steps if known :) Thanks.

[arunx2]

Hey @arunx2, thanks for this contribution! Looks good overall. I haven't tested the code yet, but I did do an initial read through it and left some comments

I pushed new changes and adopted the new API. Please review when you have time.

[arunx2]

@Jeffail @mihaitodor Can one of you kindly review the PR? Kindly let me know what are the next steps. I'm just keeping the PR up to date with the main branch as much as I can in the last 6 months. I'm at the brink of giving up. As I'm working with many customers in OpenSearch, I hope I can show a great demo to them with Benthos and OpenSearch. Thanks!

[is3ka1]

Hi @arunx2 ,

I think it need to make a minor adjustment of adding _ "github.com/benthosdev/benthos/v4/public/components/opensearch" import to the public/components/all/package.go file, so that cmd/benthos/main.go cli can include the opensearch output.

Also appreciate the efforts to keep the opensearch output feature usable. As someone who also needs this feature, I really hope this pr will be merged soon.

Thanks for all the work

[arunx2]

Hi @arunx2 ,

I think it need to make a minor adjustment of adding _ "github.com/benthosdev/benthos/v4/public/components/opensearch" import to the public/components/all/package.go file, so that cmd/benthos/main.go cli can include the opensearch output.

Also appreciate the efforts to keep the opensearch output feature usable. As someone who also needs this feature, I really hope this pr will be merged soon.

Thanks for all the work

Thanks @is3ka1, for your review and hope. I added the OpenSearch package to all/package.go

[arunx2]

@Jeffail @mihaitodor, Can you comment on the status/plan, please?

[Jeffail]

Hey @arunx2 it looks as though you're still using the older plugin APIs and config structs to register the plugin so before merging this we'd need to manually migrate it ourselves. New component plugins shouldn't generally require any imports from the internal package (with the exception being the inner aws package in this case). This is something we can potentially do for you but it's not a priority over other things going on right now.

[piclemx]

Do we know when this will be retaken?

[Jeffail]

The existing elasticsearch output is now using the new APIs so you can mostly just copy/paste it: https://github.com/benthosdev/benthos/blob/main/internal/impl/elasticsearch/output.go

[Jeffail]

Hey @arunx2, I finally got around to working on this, sorry for the wait. It's been merged via https://github.com/benthosdev/benthos/commit/db914ab6de1c6836e0987c95059137c2407dbc48. I made a few changes to the original design as I want to deviate away from the limitations of the original elasticsearch output.

[piclemx]

@Jeffail what are those changes?

I wanted to see if we could make any modification at some point on the error handling from OS.

[Jeffail]

@piclemx take a look at my follow up commit: https://github.com/benthosdev/benthos/commit/0b25d042b73ec1bd1bfe0d25db0d00eef45c76b5

[piclemx]

yeah, I saw it after commenting sorry 😂