Switch to the new version of OLTPBench.

[lmwnshn]

Heading

Switch to the new version of OLTPBench.

Description

Blocked on merging:

• [x] https://github.com/cmu-db/noisepage/pull/1631
• [x] https://github.com/oltpbenchmark/oltpbench/pull/384

Does NOT need OLTPBench itself to merge oltpbench_tim into master.

[noisepage-checks[bot]]

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-6.16%	tpcc	RAM disk	Details master tps=22847.0, commit tps=21440.41, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.19%	tpcc	None	Details master tps=29796.25, commit tps=29142.8, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-1.94%	tpcc	HDD	Details master tps=21697.7, commit tps=21276.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.64%	tatp	RAM disk	Details master tps=6574.02, commit tps=6747.38, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-7.81%	tatp	None	Details master tps=7537.73, commit tps=6949.0, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.52%	tatp	HDD	Details master tps=6546.74, commit tps=6907.92, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

[noisepage-checks[bot]]

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

[lmwnshn]

benchbase

[noisepage-checks[bot]]

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-54.65%	tpcc	RAM disk	Details master tps=22928.11, commit tps=10398.17, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-60.3%	tpcc	None	Details master tps=29712.44, commit tps=11794.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-57.62%	tpcc	HDD	Details master tps=21952.81, commit tps=9302.72, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-30.74%	tatp	RAM disk	Details master tps=6435.38, commit tps=4456.95, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-34.53%	tatp	None	Details master tps=7211.2, commit tps=4720.9, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-32.15%	tatp	HDD	Details master tps=6349.03, commit tps=4308.11, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

[codecov[bot]]

Codecov Report

Merging #1632 (af0592c) into master (25b264e) will increase coverage by 0.01%. The diff coverage is n/a.

:exclamation: Current head af0592c differs from pull request most recent head 714253a. Consider uploading reports for the commit 714253a to get more accurate results

@@            Coverage Diff             @@
##           master    #1632      +/-   ##
==========================================
+ Coverage   81.11%   81.13%   +0.01%     
==========================================
  Files         761      761              
  Lines       55258    55258              
==========================================
+ Hits        44825    44835      +10     
+ Misses      10433    10423      -10     

Impacted Files	Coverage Δ
src/include/storage/block_access_controller.h	88.23% <0.00%> (-5.89%)	:arrow_down:
src/execution/sql/vector.cpp	85.76% <0.00%> (-0.34%)	:arrow_down:
src/include/storage/index/bplustree.h	90.98% <0.00%> (-0.11%)	:arrow_down:
src/execution/sema/sema_builtin.cpp	60.54% <0.00%> (-0.09%)	:arrow_down:
src/execution/vm/bytecode_generator.cpp	84.00% <0.00%> (+0.08%)	:arrow_up:
src/parser/postgresparser.cpp	84.99% <0.00%> (+0.14%)	:arrow_up:
src/storage/arrow_serializer.cpp	86.87% <0.00%> (+0.62%)	:arrow_up:
src/execution/sema/sema_expr.cpp	77.27% <0.00%> (+1.13%)	:arrow_up:
...self_driving/modeling/data/grouped_op_unit_data.py	46.85% <0.00%> (+1.14%)	:arrow_up:
src/storage/index/hash_index.cpp	91.13% <0.00%> (+1.26%)	:arrow_up:
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 25b264e...714253a. Read the comment docs.

[noisepage-checks[bot]]

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-51.43%	tpcc	RAM disk	Details master tps=23246.39, commit tps=11291.67, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-57.08%	tpcc	None	Details master tps=29097.48, commit tps=12488.65, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-54.54%	tpcc	HDD	Details master tps=21521.21, commit tps=9783.1, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-26.29%	tatp	RAM disk	Details master tps=6454.45, commit tps=4757.84, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-33.08%	tatp	None	Details master tps=7294.7, commit tps=4881.72, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-25.31%	tatp	HDD	Details master tps=6349.03, commit tps=4741.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

[noisepage-checks[bot]]

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-10.48%	tpcc	RAM disk	Details master tps=23070.67, commit tps=20653.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-6.35%	tpcc	None	Details master tps=30175.57, commit tps=28259.21, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-5.71%	tpcc	HDD	Details master tps=21604.76, commit tps=20370.56, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-4.84%	tatp	RAM disk	Details master tps=6501.3, commit tps=6186.63, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-14.14%	tatp	None	Details master tps=7233.86, commit tps=6210.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-3.28%	tatp	HDD	Details master tps=6262.88, commit tps=6057.52, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

[noisepage-checks[bot]]

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-9.47%	tpcc	RAM disk	Details master tps=23070.67, commit tps=20886.88, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-7.8%	tpcc	None	Details master tps=30175.57, commit tps=27821.55, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-7.08%	tpcc	HDD	Details master tps=21604.76, commit tps=20074.48, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-7.12%	tatp	RAM disk	Details master tps=6501.3, commit tps=6038.61, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-12.44%	tatp	None	Details master tps=7233.86, commit tps=6333.97, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.39%	tatp	HDD	Details master tps=6262.88, commit tps=6113.22, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

[lmwnshn]

I don't foresee a risk of merge conflicts, so this can sit out here for a while. The plan is for me and @jkosh44 to poke at BenchBase next week to do a little light profiling work.