Index deletes

[jeffa5]

• Adds a new TxDiff type that is similar to a Tx in that it takes changesets and a map name but produces a MapDiff instead of a MapHandle.
• This MapDiff works directly off the changeset given (which represents just the changes in a single transaction) and so has some simpler code related
• There is both a typed and untyped MapDiff interface.
• It also exposes deletes in the interface, e.g. in foreach, get and a new is_deleted function.
• Rather than adding a new function to the indexing::Strategy class this leverages making the transaction from the ReadOnlyStore class. This means indexes can continue to work as they did before but now the same function can be used to get a diff and execute different logic, preventing breakage

The performance impact on clients not using this extra logic is expected to be near-zero: it sets writes on the changeset and on the commits of a map. These are already constructed and so only expected to be a small overhead

• When building the TxDiff we set a new flag on the calls to build it (keep_writes)
• When doing historical queries we also set this flag so that commits when building up the map store the writes
• When building the map we always store the commit if this flag is set
• When building the changeset for the diff this flag causes us to copy the writes out from the commit and into the changeset

Fixes #4267

[ccf-bot]

index-deletes@53375 aka 20221104.38 vs main ewma over 20 builds from 53119 to 53372

Click to see table

main

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	ls_virtual_cft^	ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_jwt_virtual_cft^	hist_sgx_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
53119	20221103.3	0.773144	17146.3	43841.2	12575.3	4581.04	3658.62	3255.39	25426.6	6328.1	8.27106e+07	16641.3	1.53396e+07	6081.27	1.50774e+07	2077.81	8.78598e+06	1712.76	8.78598e+06	1653.78	7.99955e+06	830882	1.1846e+06	8.17359e+06	3.102e+07
53126	20221103.5	0.780781	17199.9	45694.4	12513.2	4428.27	3518.69	3241.99	22646.2	6371.1	8.19242e+07	16647.3	1.53396e+07	5846.69	1.56017e+07	2094.59	8.52384e+06	1709	8.78598e+06	1637.92	7.73741e+06	834361	1.1763e+06	8.1504e+06	3.06435e+07
53136	20221103.8	0.759214	17690	41986.5	12462.2	4362.39	3605.35	3408.51	23850.5	6294.53	8.24484e+07	16505.6	1.50774e+07	5942.64	1.45532e+07	2073.12	8.78598e+06	1711.12	8.52384e+06	1655.9	7.99955e+06	835934	1.1793e+06	8.14337e+06	3.09118e+07
53158	20221103.15	0.763627	17649.3	41882.6	12453.9	4197.38	3475.59	3405.41	27861.7	6365.46	8.27106e+07	16647.5	1.50774e+07	5783.26	1.50774e+07	2097.36	9.31027e+06	1710.9	8.78598e+06	1652.36	8.2617e+06	832864	1.17723e+06	8.15572e+06	3.09398e+07
53210	20221103.31	0.766492	16803.2	45640.1	12822.6	4363.88	3511.79	3331.85	23011.9	6353.47	8.27106e+07	16670.3	1.50774e+07	5868.2	1.48153e+07	2077.43	8.78598e+06	1711.41	8.52384e+06	1629.06	7.73741e+06	833023	1.17427e+06	8.15563e+06	3.15252e+07
53233	20221103.37	0.778772	17506.7	43649.5	12563	4351.9	3575.08	3267.81	23638.2	6335.48	8.21863e+07	16858.1	1.53396e+07	6133.77	1.50774e+07	2101.39	8.78598e+06	1710.07	8.52384e+06	1650.6	7.99955e+06	824931	1.18122e+06	8.17336e+06	3.08722e+07
53246	20221103.41	0.76819	17566.2	43859.1	12604.7	4257.32	3559.32	3208.48	25314.9	6419.67	8.19242e+07	16559.4	1.58639e+07	5781.16	1.45532e+07	2075.12	8.78598e+06	1716.16	8.78598e+06	1641.02	7.99955e+06	832975	1.18159e+06	8.15407e+06	3.14419e+07
53247	20221103.42	0.766581	16854.7	43412.3	12426.3	4434.72	3441.76	3202.14	23584.1	6389.19	8.29727e+07	16674.9	1.50774e+07	5834.38	1.50774e+07	2076.54	8.52384e+06	1716.95	8.78598e+06	1658.71	8.2617e+06	826318	1.181e+06	8.15014e+06	3.08438e+07
53272	20221103.49	0.771439	17714.2	45612.5	12502.5	4405.44	3637.68	3382.39	27917.7	6264.07	8.27106e+07	16586.2	1.50774e+07	5967.51	1.50774e+07	2082.95	8.78598e+06	1712.78	8.52384e+06	1657.27	7.99955e+06	832774	1.17503e+06	8.06029e+06	3.09974e+07
53289	20221103.55	0.82705	16835.7	43871	12623.6	4351.05	3551.42	3286.09	29438.9	6281.81	8.27106e+07	16665.1	1.50774e+07	5873.59	1.48153e+07	2102.08	9.31027e+06	1709.63	8.52384e+06	1653	7.99955e+06	830197	1.17981e+06	8.1565e+06	3.07268e+07
53292	20221103.56	0.764161	17345.3	41939.9	12629.2	4400	3474.57	3264.95	22683.5	6382.11	8.24484e+07	16548.8	1.53396e+07	5953.06	1.50774e+07	2094.32	8.52384e+06	1709.42	8.78598e+06	1637.89	8.2617e+06	832753	1.17847e+06	8.14599e+06	3.1879e+07
53300	20221104.2	0.770538	17239.9	41920.7	12417.9	4416.88	3588.54	3382.98	25760.8	6360.03	8.27106e+07	16651.3	1.50774e+07	5859.43	1.45532e+07	2100.61	8.52384e+06	1712.4	8.2617e+06	1655.21	8.2617e+06	827457	1.18291e+06	8.15059e+06	3.06964e+07
53306	20221104.5	0.783901	17318.8	43752	12552.9	4222.49	3402.36	3368.23	22160.2	6413.35	8.24484e+07	16657.4	1.58639e+07	6097.99	1.48153e+07	2100.62	8.52384e+06	1715.75	8.52384e+06	1652.5	8.2617e+06	839662	1.17609e+06	8.09915e+06	3.10505e+07
53320	20221104.11	0.78411	17379.8	41976.3	12460.2	4413.08	3662.51	3354.53	25957.5	6319.73	8.24484e+07	16620	1.50774e+07	6043	1.50774e+07	2092.54	9.31027e+06	1705.68	8.2617e+06	1645.25	8.2617e+06	834976	1.17878e+06	8.15085e+06	3.07595e+07
53324	20221104.13	0.772371	16960.7	43668.5	12467.1	4366.77	3439.43	3185.98	25790.7	6438.82	8.21863e+07	16831.5	1.53396e+07	6132.5	1.50774e+07	2100.8	9.04813e+06	1712.64	8.2617e+06	1646.04	7.99955e+06	839249	1.17705e+06	8.15374e+06	3.07337e+07
53332	20221104.17	0.785184	17412.8	45465.9	12602.5	4618.7	3521.11	3457.1	26388.3	6429.76	8.27106e+07	16608	1.50774e+07	6061.89	1.48153e+07	2097.76	8.52384e+06	1712.32	8.52384e+06	1655.31	8.2617e+06	827037	1.18558e+06	8.17425e+06	3.07102e+07
53338	20221104.20	0.771655	17074.8	45631.2	12568	4355.38	3627.32	3278.39	24279.2	6345.3	8.19242e+07	16489.6	1.50774e+07	5863.58	1.48153e+07	2085.84	8.78598e+06	1716.49	8.52384e+06	1650.05	7.99955e+06	830919	1.1775e+06	8.15887e+06	3.11383e+07
53341	20221104.22	0.780368	17485.7	45688.5	12439.3	4618.48	3520.54	3279.77	23717.7	6401.88	8.27106e+07	16633.4	1.50774e+07	5928.03	1.45532e+07	2101.36	9.31027e+06	1712.98	8.2617e+06	1652.65	7.99955e+06	832478	1.17769e+06	8.15413e+06	3.07203e+07
53361	20221104.32	0.778085	17277.7	45639.2	12771.5	4471.02	3617.01	3314.19	26035.4	6320.4	8.27106e+07	16636.2	1.53396e+07	6074.57	1.50774e+07	2101.3	9.31027e+06	1710.65	8.52384e+06	1653.72	7.99955e+06	828228	1.17847e+06	8.15524e+06	3.07544e+07
53372	20221104.37	0.747339	17537.9	47951.9	12500.7	4406.32	3658.88	3466.03	22156.5	6300.8	8.19242e+07	16509.2	1.53396e+07	5893.35	1.48153e+07	2084.22	8.78598e+06	1713.02	8.78598e+06	1634.8	8.2617e+06	837818	1.17568e+06	8.14794e+06	3.07581e+07

index-deletes

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	ls_virtual_cft^	ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_jwt_virtual_cft^	hist_sgx_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
53097	20221102.50	0.776877	17319	40052.5	11330.1	4613.24	3559.45	3333.63	24141.6	6337.35	8.13999e+07	12959.3	1.66503e+07	5530.3	1.6126e+07	2071.13	8.52384e+06	1705.74	8.78598e+06	1628.08	7.99955e+06	836393	1.17986e+06	8.16756e+06	3.13088e+07
53135	20221103.7	0.773664	17446	43631.9	12427	4588.42	3584.34	3497.71	23215.3	6364.29	8.27106e+07	16663.6	1.56017e+07	5928.9	1.48153e+07	2100.12	8.78598e+06	1714.14	8.52384e+06	1644.93	7.99955e+06	834681	1.17729e+06	8.16958e+06	3.07337e+07
53364	20221104.33	0.770584	17504.6	45553	12467	4347.39	3664.19	3457.81	24057.9	6379.09	8.24484e+07	16624.3	1.50774e+07	6066.11	1.50774e+07	2066.75	8.52384e+06	1708.61	8.2617e+06	1647.37	8.2617e+06	831393	1.17688e+06	8.1467e+06	3.08234e+07
53368	20221104.35	0.751305	17111.7	47761.5	14003.4	4482.71	3605.19	3338.81	27105.1	6425.55	8.27106e+07	16674	1.50774e+07	6050.91	1.50774e+07	2066.68	8.52384e+06	1706.79	8.52384e+06	1654.04	7.99955e+06	838311	1.17688e+06	8.15287e+06	3.06968e+07
53375	20221104.38	0.76042	17090.1	43804.6	12841.5	4434.29	3605.05	3456.22	24077.4	6367.54	8.24484e+07	16631.8	1.50774e+07	5826.28	1.50774e+07	2065.76	8.52384e+06	1711.18	8.52384e+06	1628.02	7.73741e+06	839045	1.17798e+06	8.15313e+06	3.12176e+07

[jeffa5]

Note: I expect the metrics for memory usage to increase as the logging sample now maintains an additional index so that we can properly test this new behaviour.