OpenSearch Performance Experiments Results

[travisbenedict]

Background

Using OpenSearch 1.2 build 762 (arm64/x64) I ran a set of ~20 performance tests for each of the following single node configurations:

1. m5.xlarge - Security enabled
2. m5.xlarge - Security disabled
3. m6g.xlarge - Security enabled
4. m6g.xlarge - Security disabled

All tests were running using OpenSearch Benchmark with an i3.8xlarge EC2 instance as the load generation host. The tests used a modified version of the default schedule for the nyc_taxis workload which runs the original schedule twice with all operations in warmup mode and three times as the standard schedule, commonly known as two warmup and three test iterations. Additional aggregations were run on the results of each test to average together metrics across different query types in order to create a set of query summary metrics.

A new load generator and new OpenSearch single node cluster were provisioned for each test.

Findings

Some random variation between tests is expected. For indexing throughput the standard deviation as a percentage of the mean of any percentile statistic, excluding p100, is about 5% across all configurations. For query latency this is about 10%.

Average latency for all queries in a workload can vary by 20% or more between any given test. Why this is will require more research. In the meantime we should avoid outright comparisons of one test to another.

Included below are some approximate statistics for index and query metrics for each configuration. This includes the expected (average) value, the standard deviation as a percentage of the mean and the percent difference between the min and max. This table is meant to give people a framework for understanding their performance test results and should not necessarily be taken as a ground truth.

Instance Type	Security	Expected Indexing Throughput Avg (req/s)	Indexing StDev% Mean	Indexing MinMax% Diff	Expected Query Latency p90 (ms)	Expected Query Latency p99 (ms)	Query StDev% Mean	Query MinMax% Diff
m5.xlarge	Enabled	30554	~5%	~12%	431	449	~10%	~23%
m5.xlarge	Disabled	34472	~5%	~15%	418	444	~10%	~25%
m6g.xlarge	Enabled	38625	~3%	~8%	497	512	~8%	~23
m6g.xlarge	Disabled	45447	~2%	~3%	470	480	~5%	~15%

Raw Data

[BlackMetalz]

Can you guide some steps on how to test @travisbenedict since I have no idea to run a benchmark with this:

https://github.com/opensearch-project/opensearch-benchmark/issues/106

[travisbenedict]

Hey @BlackMetalz

After starting OpenSearch on a node I ran the following OpenSearch Benchmark commands depending on the cluster type.

Clusters with security plugin disabled: opensearch-benchmark execute_test --pipeline=benchmark-only --workload=nyc_taxis --target-hosts=<my_endpoint> --telemetry-params=node-stats-sample-interval:60 --telemetry=node-stats --kill-running-processes --workload-repository=private

Clusters with security plugin enabled: opensearch-benchmark execute_test --pipeline=benchmark-only --workload=nyc_taxis --target-hosts=<my_endpoint> --client-options=basic_auth_user:'admin',basic_auth_password:'admin',verify_certs:false,use_ssl:true --telemetry-params=node-stats-sample-interval:60 --telemetry=node-stats --kill-running-processes --workload-repository=private

In my ~/.benchmark/benchmarks/workloads/private/ directory I had a modified version of the OpenSearch Benchmark Workloads repo which contained the test_procedure file for nyc_taxis linked above

I hope this helps. Sorry for not including more details in the first place

[layavadi]

Is there similar result for vector search for reference ?

[dblock]

This is quite old, I'm going to close this issue because it's not calling for anything actionable. Since then, we've built https://opensearch.org/benchmarks and there's a bunch of stuff there.

@layavadi I don't believe we publish vector numbers there (yet). That's https://github.com/opensearch-project/opensearch-benchmark/issues/103. However, AWS has a very detailed blog post on what vector search performance looks like in production and the tradeoffs that I think can help.