vortex
vortex copied to clipboard
Benchmarks baseline seems to be stuck in the past
Many recent benchmarks runs I've seen use the same commit as their baseline (https://github.com/vortex-data/vortex/commit/9b7f792e88aa0434b37b61138c7fbbea2032cd07, which is almost a week old), regardless of if they are rebased/updated or not.
For example, in https://github.com/vortex-data/vortex/pull/5026 it seems like some benchmarks use the right baseline (FineWeb NVMe) while others use this older one (TPC-H, Clickbench)
Examples:
- https://github.com/vortex-data/vortex/pull/4989#issuecomment-3417402832
- https://github.com/vortex-data/vortex/pull/5024
- https://github.com/vortex-data/vortex/pull/5026#issuecomment-3427516407
It was just a failing benchmark, @joseph-isaacs noted that it might make sense to split up some of the benchmarks so they can make progress independently.