[PROPOSAL] Assistance

[Magnitus-]

Hi, I just noticed that you started this project.

We started our own opensearch provider recently although api coverage is still small at the moment: https://github.com/Ferlab-Ste-Justine/terraform-provider-opensearch

I'm not sure how far you are on the implementation roadmap, but let us know if we can contribute with anything we have done so far.

[phillbaker]

Hi, thanks for the interest! Please see the discussion here: https://github.com/phillbaker/terraform-provider-elasticsearch/issues/299

The current plan is to merge the existing opensearch/opendistro resources from that provider into this repo as it has decent coverage.

I do like how you've broken out the attributes for ISM policy, instead of using just a json text blob: https://github.com/Ferlab-Ste-Justine/terraform-provider-opensearch/blob/main/provider/resource_opensearch_ism_policy.go

Perhaps we can take a look at integrating that work into this provider!

[Magnitus-]

I'm glad to hear it. I was afraid there would be a lot of opacity around the progress of this project and that we'd have to support our own provider separately for a while longer before ultimately switching.

While not a universally popular opinion I'm sure, I'm personally relieved to hear that this provider will be focused on opensearch given that increasingly, opensearch and elasticsearch will diverge (it is actually the main reason why I decided to start our own repo instead of contributing back to your repo as I was unsure of the long term prospect of supporting both in the same provider). It will keep things simpler I think.

I'll be happy to assist in any way you require with the integration of what we've done with the ism policy (or anything else really) once you are ready.

Two points that I'll mention:

• I initially tried your provider before starting our own, I had an opensearch version error with our on prem opensearch 2.2.1 (https://github.com/Ferlab-Ste-Justine/kvm-opensearch-server)
• I'm wondering what the sdk situation will be. There is an official opensearch one, but it is incomplete (ex: https://github.com/opensearch-project/opensearch-go/issues/94), otherwise I would probably have used that one my end. I guess there will be some focus on adding missing functionality to it and ultimately using it?

[phillbaker]

Sorry for the slow reply.

I had an opensearch version error with our on prem opensearch 2.2.1

Hi, yes, that's likely due to this line: https://github.com/phillbaker/terraform-provider-elasticsearch/blob/1ff79e7de71161651f5fe64d73366aab778ea663/es/provider.go#L478. We don't have testing support yet for OpenSearch 2.x and it does have breaking api features, so the provider hasn't incorporated support for it.

I'm wondering what the sdk situation will be

This is a great question. As you alluded to, there's not a great solution currently. For now, my thinking is that the sdk is an implementation detail which should not impact the external API of the provider. To that end, as things improve, we can consider replacing the current olivere/elastic client with something that's maintained and specific to opensearch.

What do you think?

[Magnitus-]

Sorry for the slow reply.

No worries.

Hi, yes, that's likely due to this line: https://github.com/phillbaker/terraform-provider-elasticsearch/blob/1ff79e7de71161651f5fe64d73366aab778ea663/es/provider.go#L478. We don't have testing support yet for OpenSearch 2.x and it does have breaking api features, so the provider hasn't incorporated support for it.

So far, I validated manually on a local terraformed libvirt/kvm setup for opensearch 2.2.1 (it has worked well enough for my solo development efforts, but I'll be the first to admit that it doesn't scale the way automated integration/functional tests do when many developers with varying amounts of meticulousness get involved).

I don't think the libvirt/kvm setup, while practically like production in our case, would be great for a testing pipeline, but we could have a more lightweight dockerized setup integrated directly into github actions.

If you point me to a shared working space, I could look into integrating that after I'm back from vacation if you like.

This is a great question. As you alluded to, there's not a great solution currently. For now, my thinking is that the sdk is an implementation detail which should not impact the external API of the provider. To that end, as things improve, we can consider replacing the current olivere/elastic client with something that's maintained and specific to opensearch.

What do you think?

Makes sense.

I started on our end implementing missing pieces of the opensearch sdk in our provider for what we needed. I'll see what adapting them to the official sdk and getting them merged will entail when I'm back.

I guess as soon as the missing api surface is implemented, we can more seriously consider swapping.