Initial end-to-end tests simulating host tampering, impact on historical queries

[eddyashton]

Part of #3692.

The current tests include truncation and removal of ledger files. I have a test that corrupts ledger files, but this fails unexpectedly - I'll investigate that separately.

I haven't yet included tests that corrupt index files. The tricky thing here is that we need to create a huge volume of transactions before the index is flushed to disk, rather than kept in-memory. I've reduced the sample app's indexing thresholds so the files are at least written, but they're not relied on until we fill all buckets. This may already be a perf hit that needs to be removed, and I think to support this properly we'll need to make the index thresholds adjustable per-node, which relies on some other work (app-specific configuration).

[ccf-bot]

e2e_malicious_host@48639 aka 20220816.5 vs main ewma over 20 builds from 48399 to 48635

Click to see table

main

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
48399	20220808.2	6499.33	8.42835e+07	20415.4	1.66503e+07	5670.06	1.53396e+07	2554.96	1.00967e+07	1630.43	1.645e+08	2118.44	9.31027e+06	1412.02	9.81771e+07	1932.81	9.04813e+06	20513.5	871966	1.36106e+06	9.20859e+06	3.56168e+07
48407	20220808.5	6438.4	8.40213e+07	20249.2	1.6126e+07	5638.84	1.53396e+07	2548	1.00967e+07	1595.51	1.67121e+08	2108.24	9.31027e+06	1430.87	9.81771e+07	1927.33	8.78598e+06	19704.5	888306	1.36669e+06	9.19206e+06	3.58669e+07
48422	20220808.12	6398.02	8.3497e+07	20137.7	1.63882e+07	5695.03	1.53396e+07	2625.18	9.83456e+06	1582.61	1.66859e+08	2120.63	8.78598e+06	1404	9.73907e+07	1935.04	9.04813e+06	19485.4	902483	1.39112e+06	9.26278e+06	3.59298e+07
48427	20220808.14	6230.03	8.40213e+07	19679.8	1.58639e+07	5547.25	1.53396e+07	2417.7	1.03588e+07	1550.66	1.645e+08	2072.83	9.04813e+06	1369.48	9.76528e+07	1959.08	9.04813e+06	17632.4	920524	1.40023e+06	9.3901e+06	3.63114e+07
48435	20220808.17	6396.62	8.37592e+07	20212.4	1.58639e+07	5676.31	1.56017e+07	2492.05	1.00967e+07	1582.1	1.66859e+08	2084.38	9.31027e+06	1401.8	9.79149e+07	1936.03	9.04813e+06	22698	907599	1.38341e+06	9.30486e+06	3.55556e+07
48452	20220809.2	6397.96	8.37592e+07	20177.9	1.63882e+07	5695.08	1.56017e+07	2498.61	1.00967e+07	1602.55	1.66597e+08	2093.11	9.31027e+06	1399.63	9.81771e+07	1931.37	8.78598e+06	20235.1	903311	1.37495e+06	9.21688e+06	3.54939e+07
48461	20220809.7	6585.9	8.3497e+07	20158	1.63882e+07	5669.79	1.53396e+07	2557.5	1.00967e+07	1626.24	1.645e+08	2121.12	9.31027e+06	1443.42	9.84392e+07	1986.42	9.04813e+06	18271.3	886722	1.41396e+06	9.41163e+06	3.59292e+07
48466	20220809.9	6444.69	8.37592e+07	20005.2	1.56017e+07	5605.49	1.56017e+07	2541.33	1.00967e+07	1618.96	1.66859e+08	2115.35	9.31027e+06	1450.97	9.81771e+07	2014.27	9.04813e+06	20835.2	901011	1.37201e+06	9.22938e+06	3.58669e+07
48469	20220810.1	6186.62	8.42835e+07	20033.2	1.58639e+07	5604.55	1.53396e+07	2556.71	9.83456e+06	1587.94	1.64762e+08	2082.75	9.04813e+06	1421.15	9.84392e+07	1944.56	9.04813e+06	22527.5	906189	1.38172e+06	9.21684e+06	3.58669e+07
48486	20220811.1	6270.83	8.40213e+07	19935.8	1.58639e+07	5612.96	1.53396e+07	2499.35	1.2456e+07	1583.15	1.66859e+08	2112.67	9.31027e+06	1405.59	9.81771e+07	1923.87	9.04813e+06	23657.6	917679	1.38602e+06	9.28794e+06	3.56788e+07
48499	20220811.7	6318.24	8.45456e+07	20081.4	1.66503e+07	5615.16	1.53396e+07	2595.81	1.00967e+07	1589.3	1.66072e+08	2084.45	9.31027e+06	1416.86	9.84392e+07	1942.12	1.11453e+07	17960.4	888152	1.37708e+06	9.23767e+06	3.56788e+07
48519	20220812.3	6254.67	8.29727e+07	20234.9	1.69124e+07	5607.59	1.56017e+07	2489.25	1.00967e+07	1591.82	1.66597e+08	2123.82	9.04813e+06	1429.55	9.81771e+07	1935.56	9.04813e+06	20089.9	898322	1.37302e+06	9.23771e+06	3.58042e+07
48526	20220812.7	6074.64	8.42835e+07	19544.3	1.66503e+07	5648.38	1.50774e+07	2609.43	1.00967e+07	1612.39	1.645e+08	2111.9	9.57242e+06	1440.93	9.66042e+07	2018.87	8.78598e+06	18707.2	921975	1.33881e+06	9.33021e+06	3.47113e+07
48562	20220815.2	6242.99	8.42835e+07	19715.5	1.63882e+07	5635.6	1.53396e+07	2482.55	1.32424e+07	1574.14	1.66597e+08	2066.59	9.04813e+06	1406.98	9.81771e+07	1916.87	9.04813e+06	19707.5	909447	1.35619e+06	9.20028e+06	3.56168e+07
48572	20220815.8	6341.7	8.48077e+07	19763.8	1.66503e+07	5496.3	1.53396e+07	2501.6	9.83456e+06	1570.93	1.66597e+08	2107.98	9.31027e+06	1423.48	9.87014e+07	1919.72	8.78598e+06	17907.1	901165	1.3418e+06	9.10214e+06	3.59292e+07
48583	20220815.13	6332.98	8.40213e+07	19911.4	1.66503e+07	5538.12	1.53396e+07	2546.68	9.83456e+06	1586.74	1.645e+08	2108.78	9.04813e+06	1406.45	9.81771e+07	1926.97	9.04813e+06	19969.9	871522	1.37035e+06	9.28794e+06	3.56174e+07
48592	20220815.17	6485.44	8.48077e+07	20376.3	1.66503e+07	5673.41	1.53396e+07	2594.85	1.00967e+07	1618.96	1.66597e+08	2112.77	9.04813e+06	1425.72	9.81771e+07	1855.64	1.14074e+07	18035.6	905909	1.37467e+06	9.32596e+06	3.57411e+07
48597	20220815.19	6322.75	8.37592e+07	20129	1.56017e+07	5646.1	1.56017e+07	2604.46	9.83456e+06	1581.47	1.66597e+08	2119.41	9.04813e+06	1407.74	9.79149e+07	1919.54	9.04813e+06	21175.9	908606	1.38631e+06	9.27116e+06	3.59298e+07
48608	20220815.24	6185.84	8.3497e+07	19745.5	1.63882e+07	5630.7	1.53396e+07	2495.22	1.00967e+07	1581.38	1.66597e+08	2126.35	9.31027e+06	1400.49	9.73907e+07	1911.7	9.04813e+06	19982.3	890280	1.39008e+06	9.35587e+06	3.59298e+07
48635	20220816.3	6284.39	8.37592e+07	20503.9	1.69124e+07	5613.67	1.53396e+07	2536.88	1.00967e+07	1600.05	1.66859e+08	2114.81	9.04813e+06	1408.05	9.81771e+07	1923.09	8.78598e+06	20592.2	908239	1.36952e+06	9.21273e+06	3.58663e+07

e2e_malicious_host

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
46343	20220610.13	6061.03	8.40213e+07	19194.7	1.69124e+07	5595.51	1.50774e+07	2537.43	1.00967e+07	1615.52	1.63189e+08	2120.37	9.04813e+06	1431.53	9.81771e+07	1886.78	8.52384e+06	15787.5	903588	1.37568e+06	9.18377e+06	3.63121e+07
46820	20220621.27	6385.32	8.40213e+07	19997.6	1.56017e+07	5489.77	1.53396e+07	2588.42	1.00967e+07	1610.92	1.63451e+08	2084.13	9.31027e+06	1399.56	9.84392e+07	1924.85	9.04813e+06	15659.6	910906	1.36233e+06	9.17559e+06	3.61831e+07
47712	20220714.15	6622.86	8.3497e+07	20339.2	1.58639e+07	5649.37	1.50774e+07	2547.3	1.00967e+07	1640.29	1.62927e+08	2126.56	9.31027e+06	1454.63	9.84392e+07	1985.91	9.04813e+06	17651.3	907876	1.37874e+06	9.26688e+06	3.61831e+07
47918	20220721.7	6442.78	8.42835e+07	20192.1	1.69124e+07	5683.82	1.53396e+07	2611.09	1.00967e+07	1602.39	1.645e+08	2132.07	9.31027e+06	1418.89	9.79149e+07	1924.71	9.04813e+06	19699.6	903550	1.38079e+06	9.38584e+06	3.56794e+07
48639	20220816.5	6194.37	8.37592e+07	20008.2	1.69124e+07	5326.3	1.53396e+07	2553.04	1.00967e+07	1614.01	1.645e+08	2152.4	9.31027e+06	1414.53	9.81771e+07	1923.85	9.04813e+06	12797.3	908884	1.3732e+06	9.24601e+06	3.50685e+07

[eddyashton]

I'm hitting occasional node crashes with file truncation, so converting this to draft while I investigate.

[achamayou]

@eddyashton that does seem especially good to have with the file copy/management changes we've discussed coming up.

[eddyashton]

Closing this PR as it has rotted, and the key work is covered by #4149.