dolt icon indicating copy to clipboard operation
dolt copied to clipboard

Remove background async index queue

Open max-hoffman opened this issue 1 year ago • 8 comments

Remove the global async iter queue that is not used by new format. This has little effect on a long-running set of tests or a server that runs multiple tests, but muddies the waters for individual benchmarking.

max-hoffman avatar Aug 10 '22 15:08 max-hoffman

It's very surprising to me that this wouldn't regress perf on large indexed joins, at least on LD_1. Can we perf test this against some turbine queries?

reltuk avatar Aug 10 '22 15:08 reltuk

It was written specifically to boost perf for old format index joins, so I'm not sure now is the time to merge this. It is pure drag on the new format though.

max-hoffman avatar Aug 10 '22 15:08 max-hoffman

#benchmark

max-hoffman avatar Aug 10 '22 17:08 max-hoffman

@max-hoffman workflow run: https://github.com/dolthub/dolt/actions/runs/2834409997

github-actions[bot] avatar Aug 10 '22 17:08 github-actions[bot]

@max-hoffman LD_1

test_name from_latency_median to_latency_median is_faster
tpcc-scale-factor-1 5033.35 4599.99 0
test_name server_name server_version tps test_name server_name server_version tps is_faster
tpcc-scale-factor-1 dolt 60fea25741d2c5d88abbc50bb3f29d4778228076 1.17 tpcc-scale-factor-1 dolt eec9c89b3ebae2108911b3623e0185823f4bf5ab 1.16 0

github-actions[bot] avatar Aug 10 '22 17:08 github-actions[bot]

@max-hoffman DOLT_1

test_name from_latency_median to_latency_median is_faster
tpcc-scale-factor-1 6135.91 5607.61 0
test_name server_name server_version tps test_name server_name server_version tps is_faster
tpcc-scale-factor-1 dolt 60fea25741d2c5d88abbc50bb3f29d4778228076 1.47 tpcc-scale-factor-1 dolt eec9c89b3ebae2108911b3623e0185823f4bf5ab 0.97 0

github-actions[bot] avatar Aug 10 '22 17:08 github-actions[bot]

@max-hoffman LD_1

read_tests from_latency_median to_latency_median is_faster
covering_index_scan 7.17 7.17 0
groupby_scan 23.1 97.55 -1
index_join_scan 25.28 25.28 0
index_scan 63.32 127.81 -1
oltp_point_select 1.1 1.1 0
oltp_read_only 11.24 11.24 0
select_random_points 1.93 1.67 1
select_random_ranges 2.07 2.07 0
table_scan 58.92 59.99 0
types_table_scan 569.67 569.67 0
write_tests from_latency_median to_latency_median is_faster
bulk_insert 0.001 0.001 0
oltp_delete_insert 23.52 23.52 0
oltp_insert 10.09 10.09 0
oltp_read_write 41.85 41.85 0
oltp_update_index 11.45 11.45 0
oltp_update_non_index 8.43 8.43 0
oltp_write_only 30.81 30.81 0
types_delete_insert 161.51 161.51 0

github-actions[bot] avatar Aug 10 '22 18:08 github-actions[bot]

@max-hoffman DOLT_1

read_tests from_latency_median to_latency_median is_faster
covering_index_scan 2.71 2.76 0
groupby_scan 24.83 24.83 0
index_join_scan 7.98 7.84 0
index_scan 43.39 44.17 0
oltp_point_select 0.5 0.51 0
oltp_read_only 8.58 8.74 0
select_random_points 0.83 0.83 0
select_random_ranges 1.23 1.23 0
table_scan 51.94 52.89 0
types_table_scan 530.08 530.08 0
write_tests from_latency_median to_latency_median is_faster
bulk_insert 0.001 0.001 0
oltp_delete_insert 11.24 11.04 0
oltp_insert 2.86 2.86 0
oltp_read_write 17.63 17.32 0
oltp_update_index 5.0 5.0 0
oltp_update_non_index 5.28 5.28 0
oltp_write_only 8.43 8.43 0
types_delete_insert 13.22 12.75 0

github-actions[bot] avatar Aug 10 '22 18:08 github-actions[bot]

Gonna close this.

timsehn avatar Sep 26 '22 18:09 timsehn