nessie
nessie copied to clipboard
Nessie GC: Add a benchmarking testcase with spark cluster
to benchmark #3421 and create a procedure for benchmarking the GC logic. We should do it before a critical change is introduced to a GC component. It depends on: https://github.com/projectnessie/nessie/issues/3764 Add a benchmarking test case with 100k commits and 10K references with EMR spark.
Idea: We may find the upper threshold of commits that are needed to be verified by analyzing the use cases that we have and finding the maximum threshold of commits that need to be served by Nessie (and therefore GC)