OpenSearch
OpenSearch copied to clipboard
opensearch-project
Remove mmap.extensions setting
Description
Removes index.store.hybrid.mmap.extensions setting in OS 3.0.
Follow up to https://github.com/opensearch-project/OpenSearch/pull/8508
Background
Currently OpenSearch code contains an explicit list of file extensions which load using mmap from hybridfs. Other file extensions default to nio. This PR flips the logic to keep a list of nio file extensions while all others default to mmap. This will prevent any future regressions in case Lucene adds new segment file type that should be using mmap.
Related Issues
Resolves https://github.com/opensearch-project/OpenSearch/issues/8297
Check List
- [x] New functionality includes testing.
- [x] All tests pass
- [x] New functionality has been documented.
- [x] New functionality has javadoc added
- [x] Commits are signed per the DCO using --signoff
- [x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Gradle Check (Jenkins) Run Completed with:
- RESULT: FAILURE :x:
- URL: https://build.ci.opensearch.org/job/gradle-check/22899/
- CommitID: a18b958ef02c91729e835e919f65c09481047720 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Gradle Check (Jenkins) Run Completed with:
- RESULT: FAILURE :x:
- URL: https://build.ci.opensearch.org/job/gradle-check/22900/
- CommitID: bdcfcef5134120dc1df832b5ef5f4912944172e1 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
Compatibility status:
Checks if related components are compatible with change 8d153ab
Incompatible components
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]
Skipped components
Compatible components
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]
![opensearch-trigger-bot[bot] avatar](https://avatars.githubusercontent.com/u/80134844?v=4 "Published by a pub.dev verified publisher"){kind=small}
Compatibility status:
Checks if related components are compatible with change 8d153ab
Incompatible components
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]
Skipped components
Compatible components
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Gradle Check (Jenkins) Run Completed with:
- RESULT: UNSTABLE :grey_exclamation:
- TEST FAILURES:
1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
- URL: https://build.ci.opensearch.org/job/gradle-check/22904/
- CommitID: 61382cd90f36e5e7c0cd0e8c9238023e4c50ba0f Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 71.80%. Comparing base (
95fe9cb) to head (4c59590). Report is 3 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #9392 +/- ##
============================================
- Coverage 71.86% 71.80% -0.07%
+ Complexity 62757 62726 -31
============================================
Files 5161 5161
Lines 294370 294338 -32
Branches 42579 42574 -5
============================================
- Hits 211548 211340 -208
- Misses 65386 65510 +124
- Partials 17436 17488 +52
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Compatibility status:
Checks if related components are compatible with change 8d153ab
Incompatible components
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]
Skipped components
Compatible components
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Gradle Check (Jenkins) Run Completed with:
- RESULT: UNSTABLE :grey_exclamation:
- TEST FAILURES:
1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
- URL: https://build.ci.opensearch.org/job/gradle-check/22906/
- CommitID: 36c0ec010d017090110dc0114f7b72ca745de6fa Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.
@dzane17 It looks like this change will break k-NN: https://github.com/opensearch-project/k-NN/blob/9702dd0ff19adc37844fd91201eb392fd2d2fc39/src/main/java/org/opensearch/knn/plugin/KNNPlugin.java#L359
Also the documentation will need to be updated: https://opensearch.org/docs/latest/search-plugins/knn/performance-tuning/#search-performance-tuning
We should probably do those things before we go forward with the removal.
Documentation-website PR: https://github.com/opensearch-project/documentation-website/pull/4857
This PR is stalled because it has been open for 30 days with no activity.
Hi @dzane17, the PR is stalled. Is this being worked upon? Feel free to reach out to maintainers for further reviews.
This PR is stalled because it has been open for 30 days with no activity.
This PR is stalled because it has been open for 30 days with no activity.
This PR is stalled because it has been open for 30 days with no activity.
@dzane17 - Can you address conflicts by merging the main branch and revive this PR? Good time to do that since we are have started thinking about 3.0.
@andrross have no objections to speed it up if you think so
Many will ignore the warning so keeping a longer migration window makes no difference.
We don't know how widely this setting is being used, may be no one uses it, giving sufficient is the only thing we could do.
@reta - Given we have had sometime since 2.10 was released and still thinking about 3.0, I am inclined to take this change for 3.0. Thoughts? cc: @andrross @dzane17
:x: Gradle check result for c7e729ed5c1e5705b4ee49760306f6961cb232e0: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
:x: Gradle check result for f67651e9208841b82a4ab1a61b05767cf362d3f0: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
@jainankitk @andrross @reta I have rebased this PR. Can we please get it reviewed and merged?
@reta - Given we have had sometime since 2.10 was released and still thinking about 3.0, I am inclined to take this change for 3.
+1 for 3.0, thanks @jainankitk
@reta - Given we have had sometime since 2.10 was released and still thinking about 3.0, I am inclined to take this change for 3.
+1 for 3.0, thanks @jainankitk
Thanks @reta.
@dzane17 - Please update the change description, and get this merged with help from @andrross or @reta or one of the other maintainers. Although I don't see any backport label, still ensure to not backport into 2.x.
@dzane17 @jainankitk This will break k-NN. We should update ensure k-NN gets updated before merging this change (either open an issue with a specific suggestion or create a PR directly).
:x: Gradle check result for bc0d3dd70ddf8b3678f47ba31032622278ba37e4: null
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
@dzane17 @jainankitk This will break k-NN. We should update ensure k-NN gets updated before merging this change (either open an issue with a specific suggestion or create a PR directly).
@navneet1v - Is this the only place KNN plugin is using this setting? Also, are there any file extensions that KNN needs to read as NIOFS? If not, MMAP is the new default unless overridden using NIOFS setting. cc: @vamshin
@dzane17 @jainankitk This will break k-NN. We should update ensure k-NN gets updated before merging this change (either open an issue with a specific suggestion or create a PR directly).
@navneet1v - Is this the only place KNN plugin is using this setting? Also, are there any file extensions that KNN needs to read as NIOFS? If not, MMAP is the new default unless overridden using NIOFS setting. cc: @vamshin
KNN don't use any files as NIOFS.
KNN don't use any files as NIOFS.
@navneet1v - Great, thanks for confirming. Can you also create PR to remove the additionalSettings method from KNN plugin for 3.0? We can merge both the PRs together into main branch.