opensearch-benchmark icon indicating copy to clipboard operation
opensearch-benchmark copied to clipboard

Increase benchmark-metrics-* index shard count

Open engechas opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe.

If the same OpenSearch metrics store is used for many tests, the document count can reach the 2B limit/shard.

Describe the solution you'd like

The number of shards used for the metrics index should be increased.

Describe alternatives you've considered

Another alternative is to create the index more frequently than monthly

engechas avatar Apr 28 '22 17:04 engechas

Posting comment from PR to this issue too:

On second thought, we should revert this change. This change potentially impacts other users since this enforces every user's datastore to use a 10 primary and 1 replicas index regardless of cluster configuration.

I encountered an issue with my personal datastore after this PR was merged in. Prior to the change, my OSB runs were writing metrics to an external OpenSearch datastore. After this change was incorporated, it encountered an issue where the number of shards (10) was not a multiple of the number of awareness attributes (3). For more details, see issue https://github.com/opensearch-project/opensearch-benchmark/issues/330. I could change the cluster settings to get around this issue but users should not have to change their clusters to meet the requirements made by OSB.

Instead of changing the default values as done in this PR, users can override the default number of primaries and replicas with datastore.number_of_replicas and datastore.number_of_shards in the datastore config (~/.benchmark/benchmark.ini). This is easy to change and prevents us from messing with the default benchmark-metrics settings that can potentially interfere with other users' setups.

IanHoang avatar Jun 13 '23 20:06 IanHoang