[Metadata Immutability] Change different indices lookup objects from array type to lists

[akolarkunnu]

Description

Changed the arrays to immutable List instances, added new versions of the getters which returns List instances.

Related Issues

Resolves #8647

Signed-off-by: Abdul Muneer Kolarkunnu [[email protected]]

Check List

• [x] New functionality includes testing.
• [x] All tests pass
• [ ] ~New functionality has been documented.~
• [x] New functionality has javadoc added
• [ ] ~API changes companion pull request created.~
• [ ] ~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~
• [x] Commits are signed per the DCO using --signoff
• [x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)~
• [ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:white_check_mark: Gradle check result for 045617fb4e2ecaedb0cf4f2dddf7d4f64f7a0082: SUCCESS

[github-actions[bot]]

:grey_exclamation: Gradle check result for 4dc93d9c3bf2e16fa16b6b8e2a97d164d2f5d27a: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 73.68421% with 5 lines in your changes missing coverage. Please review.

Project coverage is 72.22%. Comparing base (2847695) to head (c261581).

Files with missing lines	Patch %	Lines
.../cluster/metadata/IndexNameExpressionResolver.java	75.00%	1 Missing and 1 partial :warning:
...n/cluster/health/TransportClusterHealthAction.java	50.00%	1 Missing :warning:
.../admin/indices/datastream/GetDataStreamAction.java	0.00%	1 Missing :warning:
...arch/index/recovery/RemoteStoreRestoreService.java	0.00%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14723      +/-   ##
============================================
- Coverage     72.42%   72.22%   -0.20%     
+ Complexity    65775    65677      -98     
============================================
  Files          5318     5318              
  Lines        305739   305731       -8     
  Branches      44349    44349              
============================================
- Hits         221421   220826     -595     
- Misses        66184    66771     +587     
  Partials      18134    18134              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[sandeshkr419]

Thanks @akolarkunnu, changes look good to me! @andrross @shwetathareja Do you have any additional comments on this?

Note for maintainers - discussion on introducing breaking changes and introducing non-breaking changes separately for 2.x was discussed in https://github.com/opensearch-project/OpenSearch/pull/14557 Please do not back port to 2.x - we will have separate changes for that!

[andrross]

@akolarkunnu Looks good. Unfortunately there's a merge conflict now. Can you rebase?

[github-actions[bot]]

:grey_exclamation: Gradle check result for 594b8d86978a2f8f82f456b2e7099c64385afc08: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[sandeshkr419]

@akolarkunnu Thanks for addressing the comments above.

Looking at the concerns/comments regarding added regression, I suggest that you do a small benchmarking exercise to establish if we are actually improving this and not regressing it. It will also give us confidence regarding the changes that we are pushing and also help quantify the improvements (if any).

I did something similar as part of https://github.com/opensearch-project/OpenSearch/pull/7853

• Create some indices and a lot of aliases (I did 50k) on a single node cluster or local OpenSearch process.
• Enable debug logs so that you can see time taken to create cluster state
• Script out creating aliases in a loop - this will update cluster state with each alias change and you can see in debug logs time taken to create cluster state.
• Repeat the exercise with & without your change

While running your above alias script - you can run async-profiler as well to capture CPU flame graphs as well.

[akolarkunnu]

{"run-benchmark-test": "id_9"}

[github-actions[bot]]

:x: Gradle check result for b6b3d202598644a2edaff121063966c7c62f9292: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for d2289d2def7557f85bb53f24c50c4f515cbf6233: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:grey_exclamation: Gradle check result for d2289d2def7557f85bb53f24c50c4f515cbf6233: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[rishabh6788]

not sure why the benchmark approval didn't run. running again.

[rishabh6788]

{"run-benchmark-test": "id_9"}

[github-actions[bot]]

The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/856/ . Final results will be published once the job is completed.

[opensearch-ci-bot]

Benchmark Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/856/

Metric	Task	Value	Unit
Cumulative indexing time of primary shards		70.1057	min
Min cumulative indexing time across primary shards		70.1057	min
Median cumulative indexing time across primary shards		70.1057	min
Max cumulative indexing time across primary shards		70.1057	min
Cumulative indexing throttle time of primary shards		0	min
Min cumulative indexing throttle time across primary shards		0	min
Median cumulative indexing throttle time across primary shards		0	min
Max cumulative indexing throttle time across primary shards		0	min
Cumulative merge time of primary shards		36.1572	min
Cumulative merge count of primary shards		48
Min cumulative merge time across primary shards		36.1572	min
Median cumulative merge time across primary shards		36.1572	min
Max cumulative merge time across primary shards		36.1572	min
Cumulative merge throttle time of primary shards		6.32745	min
Min cumulative merge throttle time across primary shards		6.32745	min
Median cumulative merge throttle time across primary shards		6.32745	min
Max cumulative merge throttle time across primary shards		6.32745	min
Cumulative refresh time of primary shards		3.16103	min
Cumulative refresh count of primary shards		71
Min cumulative refresh time across primary shards		3.16103	min
Median cumulative refresh time across primary shards		3.16103	min
Max cumulative refresh time across primary shards		3.16103	min
Cumulative flush time of primary shards		11.9096	min
Cumulative flush count of primary shards		55
Min cumulative flush time across primary shards		11.9096	min
Median cumulative flush time across primary shards		11.9096	min
Max cumulative flush time across primary shards		11.9096	min
Total Young Gen GC time		1.109	s
Total Young Gen GC count		45
Total Old Gen GC time		0	s
Total Old Gen GC count		0
Store size		19.3781	GB
Translog size		5.12227e-08	GB
Heap used for segments		0	MB
Heap used for doc values		0	MB
Heap used for terms		0	MB
Heap used for norms		0	MB
Heap used for points		0	MB
Heap used for stored fields		0	MB
Segment count		26
Min Throughput	index-append	498.89	docs/s
Mean Throughput	index-append	527.63	docs/s
Median Throughput	index-append	524.3	docs/s
Max Throughput	index-append	550.13	docs/s
50th percentile latency	index-append	7353.25	ms
90th percentile latency	index-append	11086	ms
99th percentile latency	index-append	14045.9	ms
100th percentile latency	index-append	16734.8	ms
50th percentile service time	index-append	7355.02	ms
90th percentile service time	index-append	11083.5	ms
99th percentile service time	index-append	14045.9	ms
100th percentile service time	index-append	16734.8	ms
error rate	index-append	0	%
Min Throughput	wait-until-merges-finish	0.03	ops/s
Mean Throughput	wait-until-merges-finish	0.03	ops/s
Median Throughput	wait-until-merges-finish	0.03	ops/s
Max Throughput	wait-until-merges-finish	0.03	ops/s
100th percentile latency	wait-until-merges-finish	37826.2	ms
100th percentile service time	wait-until-merges-finish	37826.2	ms
error rate	wait-until-merges-finish	0	%
Min Throughput	default	19.94	ops/s
Mean Throughput	default	19.95	ops/s
Median Throughput	default	19.95	ops/s
Max Throughput	default	19.95	ops/s
50th percentile latency	default	6.08063	ms
90th percentile latency	default	6.57826	ms
99th percentile latency	default	7.66004	ms
100th percentile latency	default	9.04313	ms
50th percentile service time	default	5.34373	ms
90th percentile service time	default	5.72753	ms
99th percentile service time	default	6.78077	ms
100th percentile service time	default	8.11346	ms
error rate	default	0	%
Min Throughput	term	19.97	ops/s
Mean Throughput	term	19.97	ops/s
Median Throughput	term	19.97	ops/s
Max Throughput	term	19.97	ops/s
50th percentile latency	term	6.0112	ms
90th percentile latency	term	6.51996	ms
99th percentile latency	term	7.22844	ms
100th percentile latency	term	7.58838	ms
50th percentile service time	term	5.27451	ms
90th percentile service time	term	5.50458	ms
99th percentile service time	term	6.60677	ms
100th percentile service time	term	6.68588	ms
error rate	term	0	%
Min Throughput	phrase	19.93	ops/s
Mean Throughput	phrase	19.94	ops/s
Median Throughput	phrase	19.94	ops/s
Max Throughput	phrase	19.95	ops/s
50th percentile latency	phrase	6.42178	ms
90th percentile latency	phrase	6.90905	ms
99th percentile latency	phrase	7.61269	ms
100th percentile latency	phrase	7.8422	ms
50th percentile service time	phrase	5.70791	ms
90th percentile service time	phrase	5.99536	ms
99th percentile service time	phrase	6.73789	ms
100th percentile service time	phrase	7.51904	ms
error rate	phrase	0	%
Min Throughput	articles_monthly_agg_uncached	19.9	ops/s
Mean Throughput	articles_monthly_agg_uncached	19.91	ops/s
Median Throughput	articles_monthly_agg_uncached	19.91	ops/s
Max Throughput	articles_monthly_agg_uncached	19.92	ops/s
50th percentile latency	articles_monthly_agg_uncached	9.0888	ms
90th percentile latency	articles_monthly_agg_uncached	9.88849	ms
99th percentile latency	articles_monthly_agg_uncached	14.7582	ms
100th percentile latency	articles_monthly_agg_uncached	15.0167	ms
50th percentile service time	articles_monthly_agg_uncached	8.32409	ms
90th percentile service time	articles_monthly_agg_uncached	8.82522	ms
99th percentile service time	articles_monthly_agg_uncached	14.037	ms
100th percentile service time	articles_monthly_agg_uncached	14.1024	ms
error rate	articles_monthly_agg_uncached	0	%
Min Throughput	articles_monthly_agg_cached	20.02	ops/s
Mean Throughput	articles_monthly_agg_cached	20.02	ops/s
Median Throughput	articles_monthly_agg_cached	20.02	ops/s
Max Throughput	articles_monthly_agg_cached	20.02	ops/s
50th percentile latency	articles_monthly_agg_cached	3.76755	ms
90th percentile latency	articles_monthly_agg_cached	4.17283	ms
99th percentile latency	articles_monthly_agg_cached	4.52851	ms
100th percentile latency	articles_monthly_agg_cached	4.76928	ms
50th percentile service time	articles_monthly_agg_cached	2.96476	ms
90th percentile service time	articles_monthly_agg_cached	3.25665	ms
99th percentile service time	articles_monthly_agg_cached	3.43248	ms
100th percentile service time	articles_monthly_agg_cached	3.50423	ms
error rate	articles_monthly_agg_cached	0	%
Min Throughput	scroll	12.55	pages/s
Mean Throughput	scroll	12.59	pages/s
Median Throughput	scroll	12.58	pages/s
Max Throughput	scroll	12.66	pages/s
50th percentile latency	scroll	590.682	ms
90th percentile latency	scroll	608.248	ms
99th percentile latency	scroll	643.142	ms
100th percentile latency	scroll	659.807	ms
50th percentile service time	scroll	588.202	ms
90th percentile service time	scroll	605.833	ms
99th percentile service time	scroll	640.527	ms
100th percentile service time	scroll	657.443	ms
error rate	scroll	0	%

[opensearch-ci-bot]

Benchmark Baseline Comparison Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/18/

Metric	Task	Baseline	Contender	Diff	Unit
Cumulative indexing time of primary shards		75.2135	70.1057	-5.10782	min
Min cumulative indexing time across primary shard		75.2135	70.1057	-5.10782	min
Median cumulative indexing time across primary shard		75.2135	70.1057	-5.10782	min
Max cumulative indexing time across primary shard		75.2135	70.1057	-5.10782	min
Cumulative indexing throttle time of primary shards		0	0	0	min
Min cumulative indexing throttle time across primary shard		0	0	0	min
Median cumulative indexing throttle time across primary shard		0	0	0	min
Max cumulative indexing throttle time across primary shard		0	0	0	min
Cumulative merge time of primary shards		35.7792	36.1572	0.37803	min
Cumulative merge count of primary shards		47	48	1
Min cumulative merge time across primary shard		35.7792	36.1572	0.37803	min
Median cumulative merge time across primary shard		35.7792	36.1572	0.37803	min
Max cumulative merge time across primary shard		35.7792	36.1572	0.37803	min
Cumulative merge throttle time of primary shards		6.85675	6.32745	-0.5293	min
Min cumulative merge throttle time across primary shard		6.85675	6.32745	-0.5293	min
Median cumulative merge throttle time across primary shard		6.85675	6.32745	-0.5293	min
Max cumulative merge throttle time across primary shard		6.85675	6.32745	-0.5293	min
Cumulative refresh time of primary shards		3.11688	3.16103	0.04415	min
Cumulative refresh count of primary shards		69	71	2
Min cumulative refresh time across primary shard		3.11688	3.16103	0.04415	min
Median cumulative refresh time across primary shard		3.11688	3.16103	0.04415	min
Max cumulative refresh time across primary shard		3.11688	3.16103	0.04415	min
Cumulative flush time of primary shards		11.9941	11.9096	-0.08443	min
Cumulative flush count of primary shards		53	55	2
Min cumulative flush time across primary shard		11.9941	11.9096	-0.08443	min
Median cumulative flush time across primary shard		11.9941	11.9096	-0.08443	min
Max cumulative flush time across primary shard		11.9941	11.9096	-0.08443	min
Total Young Gen GC time		1.32	1.109	-0.211	s
Total Young Gen GC count		46	45	-1
Total Old Gen GC time		0	0	0	s
Total Old Gen GC count		0	0	0
Store size		19.3796	19.3781	-0.00143	GB
Translog size		5.12227e-08	5.12227e-08	0	GB
Heap used for segments		0	0	0	MB
Heap used for doc values		0	0	0	MB
Heap used for terms		0	0	0	MB
Heap used for norms		0	0	0	MB
Heap used for points		0	0	0	MB
Heap used for stored fields		0	0	0	MB
Segment count		29	26	-3
Min Throughput	index-append	472.099	498.892	26.7929	docs/s
Mean Throughput	index-append	495.051	527.631	32.5799	docs/s
Median Throughput	index-append	494.526	524.301	29.7751	docs/s
Max Throughput	index-append	516.079	550.132	34.0524	docs/s
50th percentile latency	index-append	7818.77	7353.25	-465.519	ms
90th percentile latency	index-append	11653	11086	-566.98	ms
99th percentile latency	index-append	15312.4	14045.9	-1266.49	ms
100th percentile latency	index-append	16781.8	16734.8	-46.9258	ms
50th percentile service time	index-append	7815.38	7355.02	-460.351	ms
90th percentile service time	index-append	11641.8	11083.5	-558.33	ms
99th percentile service time	index-append	15312.4	14045.9	-1266.49	ms
100th percentile service time	index-append	16781.8	16734.8	-46.9258	ms
error rate	index-append	0	0	0	%
Min Throughput	wait-until-merges-finish	0.0219732	0.0264364	0.00446	ops/s
Mean Throughput	wait-until-merges-finish	0.0219732	0.0264364	0.00446	ops/s
Median Throughput	wait-until-merges-finish	0.0219732	0.0264364	0.00446	ops/s
Max Throughput	wait-until-merges-finish	0.0219732	0.0264364	0.00446	ops/s
100th percentile latency	wait-until-merges-finish	45509.8	37826.2	-7683.59	ms
100th percentile service time	wait-until-merges-finish	45509.8	37826.2	-7683.59	ms
error rate	wait-until-merges-finish	0	0	0	%
Min Throughput	default	19.9434	19.9392	-0.00427	ops/s
Mean Throughput	default	19.952	19.9478	-0.00427	ops/s
Median Throughput	default	19.9525	19.9484	-0.00416	ops/s
Max Throughput	default	19.9594	19.9548	-0.00461	ops/s
50th percentile latency	default	7.29752	6.08063	-1.21689	ms
90th percentile latency	default	7.80291	6.57826	-1.22466	ms
99th percentile latency	default	9.2995	7.66004	-1.63946	ms
100th percentile latency	default	10.0042	9.04313	-0.96109	ms
50th percentile service time	default	6.42596	5.34373	-1.08223	ms
90th percentile service time	default	6.78458	5.72753	-1.05705	ms
99th percentile service time	default	8.33548	6.78077	-1.55471	ms
100th percentile service time	default	8.88488	8.11346	-0.77142	ms
error rate	default	0	0	0	%
Min Throughput	term	19.9611	19.9653	0.0042	ops/s
Mean Throughput	term	19.9663	19.97	0.00368	ops/s
Median Throughput	term	19.9668	19.9705	0.00364	ops/s
Max Throughput	term	19.971	19.974	0.00297	ops/s
50th percentile latency	term	6.92073	6.0112	-0.90954	ms
90th percentile latency	term	7.29687	6.51996	-0.77691	ms
99th percentile latency	term	8.15277	7.22844	-0.92433	ms
100th percentile latency	term	8.58024	7.58838	-0.99187	ms
50th percentile service time	term	6.10219	5.27451	-0.82767	ms
90th percentile service time	term	6.2615	5.50458	-0.75692	ms
99th percentile service time	term	7.6595	6.60677	-1.05273	ms
100th percentile service time	term	7.8381	6.68588	-1.15222	ms
error rate	term	0	0	0	%
Min Throughput	phrase	19.9535	19.9289	-0.02461	ops/s
Mean Throughput	phrase	19.9601	19.939	-0.02113	ops/s
Median Throughput	phrase	19.9605	19.9394	-0.02111	ops/s
Max Throughput	phrase	19.9655	19.9471	-0.01838	ops/s
50th percentile latency	phrase	7.24736	6.42178	-0.82558	ms
90th percentile latency	phrase	7.70502	6.90905	-0.79597	ms
99th percentile latency	phrase	9.3764	7.61269	-1.76371	ms
100th percentile latency	phrase	9.57885	7.8422	-1.73665	ms
50th percentile service time	phrase	6.45853	5.70791	-0.75062	ms
90th percentile service time	phrase	6.64824	5.99536	-0.65288	ms
99th percentile service time	phrase	8.25055	6.73789	-1.51266	ms
100th percentile service time	phrase	8.81872	7.51904	-1.29968	ms
error rate	phrase	0	0	0	%
Min Throughput	articles_monthly_agg_uncached	19.9253	19.897	-0.02829	ops/s
Mean Throughput	articles_monthly_agg_uncached	19.9361	19.9117	-0.02442	ops/s
Median Throughput	articles_monthly_agg_uncached	19.9366	19.9122	-0.02446	ops/s
Max Throughput	articles_monthly_agg_uncached	19.9449	19.9239	-0.02104	ops/s
50th percentile latency	articles_monthly_agg_uncached	10.1364	9.0888	-1.0476	ms
90th percentile latency	articles_monthly_agg_uncached	10.6082	9.88849	-0.7197	ms
99th percentile latency	articles_monthly_agg_uncached	14.6775	14.7582	0.0807	ms
100th percentile latency	articles_monthly_agg_uncached	15.021	15.0167	-0.00436	ms
50th percentile service time	articles_monthly_agg_uncached	9.28364	8.32409	-0.95955	ms
90th percentile service time	articles_monthly_agg_uncached	9.59538	8.82522	-0.77016	ms
99th percentile service time	articles_monthly_agg_uncached	14.0754	14.037	-0.0384	ms
100th percentile service time	articles_monthly_agg_uncached	14.3161	14.1024	-0.21364	ms
error rate	articles_monthly_agg_uncached	0	0	0	%
Min Throughput	articles_monthly_agg_cached	20.0165	20.0171	0.00059	ops/s
Mean Throughput	articles_monthly_agg_cached	20.0192	20.02	0.00076	ops/s
Median Throughput	articles_monthly_agg_cached	20.0189	20.0198	0.00086	ops/s
Max Throughput	articles_monthly_agg_cached	20.0225	20.023	0.0005	ops/s
50th percentile latency	articles_monthly_agg_cached	4.2238	3.76755	-0.45624	ms
90th percentile latency	articles_monthly_agg_cached	4.59667	4.17283	-0.42384	ms
99th percentile latency	articles_monthly_agg_cached	4.99173	4.52851	-0.46322	ms
100th percentile latency	articles_monthly_agg_cached	5.08646	4.76928	-0.31718	ms
50th percentile service time	articles_monthly_agg_cached	3.42228	2.96476	-0.45752	ms
90th percentile service time	articles_monthly_agg_cached	3.61051	3.25665	-0.35387	ms
99th percentile service time	articles_monthly_agg_cached	3.99348	3.43248	-0.561	ms
100th percentile service time	articles_monthly_agg_cached	4.13392	3.50423	-0.62968	ms
error rate	articles_monthly_agg_cached	0	0	0	%
Min Throughput	scroll	12.5452	12.5527	0.0075	pages/s
Mean Throughput	scroll	12.5744	12.5867	0.01233	pages/s
Median Throughput	scroll	12.5677	12.5789	0.01121	pages/s
Max Throughput	scroll	12.6341	12.6565	0.0224	pages/s
50th percentile latency	scroll	772.209	590.682	-181.526	ms
90th percentile latency	scroll	798.667	608.248	-190.419	ms
99th percentile latency	scroll	864.069	643.142	-220.928	ms
100th percentile latency	scroll	881.163	659.807	-221.356	ms
50th percentile service time	scroll	769.727	588.202	-181.525	ms
90th percentile service time	scroll	796.039	605.833	-190.207	ms
99th percentile service time	scroll	861.781	640.527	-221.254	ms
100th percentile service time	scroll	879.169	657.443	-221.727	ms
error rate	scroll	0	0	0	%

[opensearch-trigger-bot[bot]]

This PR is stalled because it has been open for 30 days with no activity.

[github-actions[bot]]

:white_check_mark: Gradle check result for ed9e85e82dada36671463e243b6b4c6e84274aa3: SUCCESS

[github-actions[bot]]

:x: Gradle check result for a7607416d34212583913ed26c57088fe21cf2d9a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Thanks for sticking with us @akolarkunnu! The DCO check is failing, I think related to the fact that you've got 354 commits in this PR. I'd suggest squashing/rebasing everything to just your single commit on top of the latest commit on main. Also, you're currently signing your commit with [email protected]. Please take a look at our DCO policy. We'd appreciate it if you could sign the commit with a valid email address.

[akolarkunnu]

Thanks for sticking with us @akolarkunnu! The DCO check is failing, I think related to the fact that you've got 354 commits in this PR. I'd suggest squashing/rebasing everything to just your single commit on top of the latest commit on main. Also, you're currently signing your commit with [email protected]. Please take a look at our DCO policy. We'd appreciate it if you could sign the commit with a valid email address.

Corrected.

[github-actions[bot]]

:white_check_mark: Gradle check result for c2615810327da6a54bc1ed9cb364be22ffe6c61e: SUCCESS