redis-benchmarks-specification icon indicating copy to clipboard operation
redis-benchmarks-specification copied to clipboard

[BUG]: Failures in benchmarking tests that use dataset configured in dbconfig

Open odidev opened this issue 1 year ago • 2 comments

Describe the bug

I am executing benchmarking tests on the Linux/AMD64 AWS instance.

I am facing issues in all the tests that have dataset configured in dbconfig. For example, consider the test “memtier_benchmark-1key-geo-60M-elements-geohash-pipeline-10.yml”. I ran the test with dataset, following the command below:

redis-benchmarks-spec-client-runner --db_server_host localhost --db_server_port 6379 --client_aggregated_results_folder ./test --flushall_on_every_test_start --run-tests-with-dataset –test memtier_benchmark-1key-geo-60M-elements-geohash-pipeline-10.yml 

The failure logs are as below:

2023-06-27 10:43:03 ERROR The total numbers of keys in setup does not match the expected spec: 1!=0. Aborting... 
2023-06-27 10:43:03 CRITICAL Some unexpected exception was caught during local work. Failing test.... 

I found a similar issue here: < https://github.com/redis/redis-benchmarks-specification/issues/206 >, and the solution was this PR: < https://github.com/redis/redis-benchmarks-specification/pull/213/files >.

Following the above PR, I edited the “keyspacelen” from 1 to 0, and the test passes successfully.

Similarly, there are 8 more tests in the suite, that use dataset. Below is the list:

memtier_benchmark-1key-geo-60M-elements-geohash.yml memtier_benchmark-1key-geo-60M-elements-geopos-pipeline-10.yml memtier_benchmark-1key-geo-60M-elements-geopos.yml memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-bybox.yml memtier_benchmark-1key-geo-60M-elements-geodist-pipeline-10.yml
memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-pipeline-10.yml memtier_benchmark-1key-geo-60M-elements-geodist.yml
memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat.yml

All the above-mentioned 9 tests use dataset and passes successfully after editing “keyspacelen” from 1 to 0.

Do you think it is reasonable to edit the “keyspacelen” to 0? If yes, I will raise a PR for the same.

Kindly let me know your thoughts on the above.

Environment (please complete the following information):

  • OS: Ubuntu 22.04
  • PyPi version package pip 23.1.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
  • Platfrom x86_64

odidev avatar Jun 27 '23 11:06 odidev