[META] Breaking changes in 3.0

[reta]

This is a meta issue which lists down all breaking changes in OpenSearch 3.0

Deprecated code in 3.0+

• [ ] Cleanup deprecated thread pool settings https://github.com/opensearch-project/OpenSearch/issues/2595
• [ ] Replace 'whitelist' terminology in Java API https://github.com/opensearch-project/OpenSearch/issues/1683
• [ ] Removed deprecated methods from JodaCompatibleZonedDateTime https://github.com/opensearch-project/OpenSearch/pull/3346
• [ ] Rename Class ending with Plugin to Module under modules dir https://github.com/opensearch-project/OpenSearch/pull/4042
• [ ] https://github.com/opensearch-project/OpenSearch/pull/4459
• [ ] #7936
• [x] #6595
• [ ] https://github.com/opensearch-project/OpenSearch/issues/3412 (https://github.com/opensearch-project/OpenSearch/pull/10625)
• [ ] https://github.com/opensearch-project/OpenSearch/pull/10894
• [ ] https://github.com/opensearch-project/OpenSearch/pull/11811 - content limits in XContent / Jackson
• [ ] https://github.com/opensearch-project/OpenSearch/pull/11670

Breaking behavior in 3.0+:

• [ ] https://github.com/opensearch-project/OpenSearch/pull/15039
• [ ] https://github.com/opensearch-project/OpenSearch/pull/15016

Removed code in 3.0+

• [ ] https://github.com/opensearch-project/OpenSearch/issues/2773
• [ ] https://github.com/opensearch-project/OpenSearch/pull/9392

Major updates:

• [ ] https://github.com/opensearch-project/OpenSearch/pull/13988 (https://github.com/opensearch-project/OpenSearch/issues/11550)
• [ ] https://github.com/opensearch-project/OpenSearch/issues/10745
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11415
• [ ] https://github.com/opensearch-project/OpenSearch/issues/14011

[reta]

@saratvemulapalli fyi, meta for 3.0 breaking changes, thanks!

[saratvemulapalli]

@CEHENKLE could you help add this to 3.0 roadmap?

[CEHENKLE]

Done.

[saratvemulapalli]

Looks like this is getting stale. @andrross /@reta do you think we can use Changelog for this ?

[reta]

@andrross /@reta do you think we can use Changelog for this ?

We probably would have to, but what are other breaking changes to mention? I think @nknize is doing some large refactorings but so far I think it was non breaking.

[andrross]

I did some searching and couldn't find any open issues, but we have deprecation warning about blocking direct access to system indices. @reta is implementing the block for direct access to system indices something we should track for 3.0?

This came up in discussion in #7778.

[reta]

@andrross I think we should track it, thanks!

[andrross]

Created issue #7936 and added to the list here. Thanks!

[msfroh]

@reta -- is this a good place to build a wishlist for breaking changes in 3.0? Do you know if we have an existing good place to do that?

(For example, https://github.com/opensearch-project/OpenSearch/issues/1029 is a longstanding issue that would be nice to address in a 3.0 release, since otherwise I think it would need to wait for 4.0. I don't want to clutter up this issue with more wishlist items, but that one would be near the top of my list.)

[reta]

@reta -- is this a good place to build a wishlist for breaking changes in 3.0? Do you know if we have an existing good place to do that?

@msfroh I think this is the designated place for breaking changes which are going to be in 3.0, regarding wishlist - I don't think we have any lists for that besides targeting issues / pull requests as v3.0.0

[jed326]

Another major update: https://github.com/apache/lucene/pull/13337 It should be non-breaking though as it's just introducing a new interface method.

[reta]

Another major update: apache/lucene#13337

Thanks @jed326 ! https://github.com/opensearch-project/OpenSearch/issues/11415 would be a better place to mention it I think, thanks!

[Bukhtawar]

I think there are few decision points that needs a broader discussion like

1. Search semantics: While async search continues as a plugin, we wanted to unify this as a mode in the mainstream search API, which might cause search response to return just an id, to be subsequently polled for final results
2. Replication semantics : We support two replication modes docrep and segrep, going forward should we consider making segrep the default replication type?
3. Feature defaults : A lot of resiliency features that have been released, including back-pressure, cluster manager throttling are disabled by default, should we consider enabling them by defaults
4. Guardrails enforcements: We have quite a few knobs to control queries, shard count but don't have hard limits enforced which might cause cluster instability

[mgodwan]

In line with the themes highlighted above, I think there are a few issues which we can discuss:

1. https://github.com/opensearch-project/OpenSearch/issues/13744 [The new way provides faster parsing and efficient way to pick parser during the document parsing step of indexing flow. Some users were relying on internal details due to which this was kept as an opt-in and experimental. We should benefit from removing this in 3.0]
2. https://github.com/opensearch-project/OpenSearch/issues/12185 [Field Data feature for _id field results in more problems than it is able to solve and it may be a good idea to remove this.
3. https://github.com/opensearch-project/OpenSearch/issues/16980#issuecomment-2581927518 [There are inconsistencies between using single update and update within a bulk operation which leads to difficulty for users as well as operators in debugging. With 3.0, we can choose one code flow which works same across both the APIs]

[vikasvb90]

With In-place Shard Split, parent shard ids will not longer exist and therefore, looping from 0 to number of shards in places such as this will no longer be valid.

This will be a breaking change because it is an assumption today that number of shards in an index cannot change dynamically. This assumption will no longer hold true after this.

Solution An iterator from IndexMetadata holding and hiding context of which shards exist can be exposed to loop over shard ids. This will be adapted in OpenSearch, plugins within OpenSearch repo and external plugins. Plugins which work directly with shards may need to adapt to this change. CCR is one such plugin.

[shwetathareja]

We should look into deprecating

1. scroll api
2. legacy templates

[raguraman86]

OPenSearch 3.x support ElasticSearch compatability?

[andrross]

OPenSearch 3.x support ElasticSearch compatability?

@raguraman86 If you're looking to migrate from Elasticsearch to OpenSearch, please take a look at the migration assistant. These tools offer support for all versions of OpenSearch and will support 3.0 as well.

[andrross]

3.0 has been released. Closing this issue.