benchto
benchto copied to clipboard
Support more run modes
We shall support at least these run modes:
-
Power Run one query at a time, with the pre-designated query order. For example, TPCH requires the Power test run 1 stream of 22 queries, with the query order
14 2 9 20 6 17 18 8 21 13 3 22 16 4 11 15 1 10 19 5 7 12
. In regular tracking, we may want to do cold/warm runs, so each query would be runruns
times, with the first execution being cold, the rest being warm. We will take the last warm execution as the reporting query. Before the first execution of each query, we need to run macros to clean the caches. E.g. ifruns = 3
, then the queries would be runclean caches, 14(cold), 14, 14(warm), clean caches, 2(cold), 2, 2(warm)...
-
Throughput Run
s
streams concurrently, and each stream needs to follow the specified query orders. e.g. See https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf for the query orders for TPCH tests. In Throughput mode, we usually don't do the cold/warm runs as in Power phase, but rather once per query. There can be a pre-warm run before the real run, usually being the Power run. So theruns
parameter can actually be neglected. -
Concurrent Run many queries using a pre-defined query concurrency(threads), e.g. 1000 queries run on 40 threads. Each thread just pick up a random query from the query set. Each query is executed
runs
times but the order shall be randomized. Usually theruns
= 1. This is to simulate a real workload on customers' clusters. Note that we will need to be able to supply randomly chosen queries from the query history, if we have any. -
Concurrent-Repeating This is what https://github.com/prestodb/benchto/pull/68 does. With pre-defined concurrency level, each thread executes the same listed queries with
runs
times. With that PR each query would be executedruns * concurrency
times. This mode is usually used if we want to run a single query on all threads for a period of time, possibly to reproduce some bugs.
The current implementation supports 1 and 4 but not 2 and 3. We shall support all 4 modes.
cc @upbram123