[BUG] Tenant Calculation(?) quite expensive

[HenryTheSir]

What is the bug? After Upgrading from 2.18 to 2.19 we tried to drop our custom build security plugin which would return true as soon as it could determine that a user has all_access.

I'm currently unsure if this is a bottleneck of the optimized priviledge evaluation or a different root cause which occurs as we have arround 250 Tenants.

How can one reproduce the bug? Steps to reproduce the behavior:

1. Create a high amount of tenants
2. Try to Index a huge amount of data
3. Observe slow ingestion and coordination nodes hot_threads showing attached thread stack

What is the expected behavior? In a write requests tenants should not be evaluated.

   22.0% (109.9ms out of 500ms) cpu usage by thread 'opensearch[coordination-node][transport_worker][T#5]'
     10/10 snapshots sharing following 130 elements
       org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration.getCEntries(SecurityDynamicConfiguration.java:288)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.lambda$mapTenants$2(ConfigModelV7.java:1149)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder$$Lambda/0x00007fd2e9413748.accept(Unknown Source)
       [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
       [email protected]/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
       [email protected]/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
       com.google.common.collect.CollectSpliterators$1.lambda$forEachRemaining$1(CollectSpliterators.java:128)
       com.google.common.collect.CollectSpliterators$1$$Lambda/0x00007fd2e9416d88.accept(Unknown Source)
       [email protected]/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
       com.google.common.collect.CollectSpliterators$1.forEachRemaining(CollectSpliterators.java:128)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator.lambda$forEachRemaining$1(CollectSpliterators.java:377)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator$$Lambda/0x00007fd2e9413dc0.accept(Unknown Source)
       [email protected]/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator.forEachRemaining(CollectSpliterators.java:373)
       [email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
       [email protected]/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
       [email protected]/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
       [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
       [email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
       [email protected]/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.mapTenants(ConfigModelV7.java:1135)
       org.opensearch.security.securityconf.ConfigModelV7.mapTenants(ConfigModelV7.java:1285)
       org.opensearch.security.privileges.PrivilegesEvaluatorImpl.mapTenants(PrivilegesEvaluatorImpl.java:598)
       org.opensearch.security.privileges.PrivilegesEvaluatorImpl.evaluate(PrivilegesEvaluatorImpl.java:520)
       org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
       org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:166)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)