arkouda icon indicating copy to clipboard operation
arkouda copied to clipboard

Update new benchmarks

Open stress-tess opened this issue 11 months ago • 1 comments

Similar to #3050

We need to go through each files in benchmarks_v2 and verify it is benchmarking the same thing as the original. This is far more nuanced than updating the tests. sort-cases in particular i remember being concerned about since it doesn't lend itself well to the new format (since it makes use of python generators, see https://github.com/Bears-R-Us/arkouda/pull/2276)

One quick litmus test is to run the new and old benchmark and make sure the results are fairly similar

  • [ ] Update aggregate_benchmark
  • [ ] Update argsort_benchmark
  • [ ] Update array_create_benchmark
  • [ ] Update array_transfer_benchmark
  • [ ] Update bigint_bitwise_binops_benchmark
  • [ ] Update coragsort_benchmark
  • [ ] Update conftest_benchmark
  • [ ] Update dataframe_indexing_benchmark
  • [ ] Update encoding_benchmark
  • [ ] Update flatten_benchmark
  • [ ] Update gather_benchmark
  • [ ] Update groupby_benchmark
  • [ ] Update in1d_benchmark
  • [ ] Update io_benchmark
  • [ ] Update no_op_benchmark
  • [ ] Update reduce_benchmark
  • [ ] Update scan_benchmark
  • [ ] Update scatter_benchmark
  • [ ] Update setops_benchmark
  • [ ] Update sort_cases_benchmark
  • [ ] Update str_locality_benchmark
  • [ ] Update stream_benchmark
  • [ ] Update substring_search_benchmark

@Bears-R-Us/arkouda-core-dev while going through these, we should keep in mind if it makes sense for something to be used as benchmark. Ideally benchmarks are capturing the core functionality that underpins most workflows. Unlike tests, I don't think we should try to benchmark as much of our functions as possible, but to focus on the ones that we really want to know if there's a performance drop off. I think we should keep this targeted to the ones we really care about, but I'm open to other opinions!

I know some of these benchmarks were added as one off to see how a newly added function performed and to track it as we optimized

Once we are confident these work and cover what we are looking for, we should work with @hokiegeek2 to use the JSON(?) output to create a grafana dashboard. Then we should work with @bmcdonald3 or @jeremiah-corrado to create a script that turns the output of these into something that is readable by the chpl nightly graphs

stress-tess avatar Mar 19 '24 18:03 stress-tess

I chatted offline with @bmcdonald3 and it seems like converting from the JSON to be readable by the chpl graphs would be quite difficult. I think the better approach is to find a way to add an option for our current benchmarks to give JSON output

stress-tess avatar Mar 19 '24 23:03 stress-tess