CCF
CCF copied to clipboard
microsoft
Test
app@51328 aka 20221017.5 vs main ewma over 20 builds from 51011 to 51323
Click to see table
main
| build_id | build_number | tpcc_virtual_cft^ | tpcc_virtual_cft_mem | ls_virtual_cft^ | ls_virtual_cft_mem | ls_jwt_virtual_cft^ | ls_jwt_virtual_cft_mem | ls_js_virtual_cft^ | ls_js_virtual_cft_mem | ls_full_js_virtual_cft^ | ls_full_js_virtual_cft_mem | ls_js_jwt_virtual_cft^ | ls_js_jwt_virtual_cft_mem | hist_sgx_cft^ | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | ls_sgx_cft^ | ls_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 51011 | 20221007.12 | 11187.7 | 0 | 40597.2 | 0 | 10412.8 | 0 | 4314.43 | 0 | 3399.11 | 0 | 3294.19 | 0 | 25389.5 | 6235.86 | 8.37592e+07 | 19821 | 1.69124e+07 | 5548.36 | 1.56017e+07 | 2523.87 | 9.57242e+06 | 2059.07 | 9.31027e+06 | 1906.84 | 9.04813e+06 | 875619 | 1.37458e+06 | 9.04985e+06 | 3.58669e+07 |
| 51025 | 20221007.17 | 11467.1 | 0 | 41236.5 | 0 | 10121.7 | 0 | 4527.82 | 0 | 3615.67 | 0 | 3274.8 | 0 | 24578.3 | 6310.28 | 8.32349e+07 | 19793.5 | 1.71746e+07 | 5621.17 | 1.53396e+07 | 2419.48 | 9.83456e+06 | 2063.54 | 9.57242e+06 | 1918.97 | 9.04813e+06 | 895804 | 1.35574e+06 | 9.19615e+06 | 3.56174e+07 |
| 51036 | 20221007.21 | 11072.4 | 0 | 42141.6 | 0 | 10163.2 | 0 | 4412.54 | 0 | 3630.63 | 0 | 3431.49 | 0 | 25969.5 | 6290.72 | 8.24484e+07 | 19773.6 | 1.6126e+07 | 5628.41 | 1.58639e+07 | 2415 | 9.57242e+06 | 2058.22 | 9.31027e+06 | 1874.55 | 9.04813e+06 | 905347 | 1.36952e+06 | 9.19206e+06 | 3.56788e+07 |
| 51053 | 20221010.2 | 11798 | 0 | 42227.8 | 0 | 10245.4 | 0 | 4300.08 | 0 | 3599.97 | 0 | 3431.49 | 0 | 30481.2 | 6305.13 | 8.29727e+07 | 19794.7 | 1.69124e+07 | 5606.72 | 1.53396e+07 | 2531.31 | 9.57242e+06 | 2067.16 | 9.31027e+06 | 1956.25 | 9.31027e+06 | 906832 | 1.35853e+06 | 9.20445e+06 | 3.55556e+07 |
| 51060 | 20221010.5 | 11225.8 | 0 | 40998.7 | 0 | 10243.7 | 0 | 4262.26 | 0 | 3544.72 | 0 | 3369.34 | 0 | 26450.9 | 5735.1 | 8.3497e+07 | 19764 | 1.6126e+07 | 5607.19 | 1.6126e+07 | 2441.86 | 1.14074e+07 | 2106.7 | 9.04813e+06 | 1955.23 | 9.04813e+06 | 925351 | 1.35297e+06 | 9.16327e+06 | 3.56174e+07 |
| 51070 | 20221010.9 | 10742 | 0 | 42747.9 | 0 | 10320.7 | 0 | 4674.44 | 0 | 3571.41 | 0 | 3396.5 | 0 | 28907 | 6480.7 | 8.29727e+07 | 19954.2 | 1.66503e+07 | 5627.41 | 1.53396e+07 | 2553.85 | 9.57242e+06 | 2111.12 | 9.31027e+06 | 1947.45 | 9.04813e+06 | 912656 | 1.38126e+06 | 9.15921e+06 | 3.58669e+07 |
| 51073 | 20221010.10 | 11037.5 | 0 | 42007.3 | 0 | 10311.2 | 0 | 4644.78 | 0 | 3545.97 | 0 | 3330.9 | 0 | 28710.5 | 6238.01 | 8.37592e+07 | 19572.5 | 1.6126e+07 | 5580.73 | 1.56017e+07 | 2371.74 | 9.83456e+06 | 2032.18 | 9.31027e+06 | 1870.01 | 9.04813e+06 | 886611 | 1.35065e+06 | 9.07391e+06 | 3.53097e+07 |
| 51085 | 20221010.15 | 11316.8 | 0 | 42993.9 | 0 | 10336.3 | 0 | 4529.17 | 0 | 3535.49 | 0 | 3433.88 | 0 | 29176.9 | 6167.23 | 8.29727e+07 | 19214.9 | 1.71746e+07 | 5530.5 | 1.53396e+07 | 2368.9 | 9.83456e+06 | 2037.46 | 9.04813e+06 | 1876.41 | 9.04813e+06 | 892875 | 1.3806e+06 | 9.45954e+06 | 3.63766e+07 |
| 51108 | 20221011.2 | 11535.3 | 0 | 39052.9 | 0 | 10195.2 | 0 | 4591.11 | 0 | 3519.27 | 0 | 3402.86 | 0 | 30358.8 | 6323.44 | 8.32349e+07 | 19655.7 | 1.6126e+07 | 5603.65 | 1.56017e+07 | 2423.14 | 9.57242e+06 | 2041.8 | 9.31027e+06 | 1873.74 | 9.57242e+06 | 887882 | 1.37735e+06 | 9.15913e+06 | 3.58669e+07 |
| 51130 | 20221011.11 | 11002.5 | 0 | 42292.9 | 0 | 10242.9 | 0 | 4382.07 | 0 | 3517.21 | 0 | 3370.72 | 0 | 25816 | 6149.71 | 8.29727e+07 | 19508.1 | 1.6126e+07 | 5317.16 | 1.53396e+07 | 2429.71 | 9.57242e+06 | 2032.13 | 9.57242e+06 | 1872.97 | 9.04813e+06 | 907836 | 1.36351e+06 | 9.22934e+06 | 3.58663e+07 |
| 51135 | 20221011.13 | 11287.9 | 0 | 42237.6 | 0 | 10684.8 | 0 | 4410.79 | 0 | 3463.51 | 0 | 3344.84 | 0 | 23655.5 | 6958.2 | 8.29727e+07 | 19881 | 1.6126e+07 | 5681.47 | 1.58639e+07 | 2416.68 | 9.83456e+06 | 2067.03 | 9.31027e+06 | 1876.73 | 9.04813e+06 | 903550 | 1.40881e+06 | 9.25014e+06 | 3.58036e+07 |
| 51144 | 20221012.3 | 10963.2 | 0 | 41097.8 | 0 | 10779.3 | 0 | 4559.6 | 0 | 3446.26 | 0 | 3353.22 | 0 | 23647.6 | 6237.84 | 8.29727e+07 | 19582.1 | 1.63882e+07 | 5550.91 | 1.53396e+07 | 2428.68 | 9.57242e+06 | 2027.51 | 9.31027e+06 | 1885.21 | 9.57242e+06 | 894084 | 1.37782e+06 | 9.45954e+06 | 3.66369e+07 |
| 51154 | 20221012.8 | 10945.8 | 0 | 43218.4 | 0 | 10674.1 | 0 | 4400.72 | 0 | 3490.57 | 0 | 3225.78 | 0 | 31361.3 | 6310.17 | 8.29727e+07 | 19588.8 | 1.58639e+07 | 5573.93 | 1.53396e+07 | 2521.39 | 9.57242e+06 | 2024.98 | 9.57242e+06 | 1871.9 | 9.04813e+06 | 911594 | 1.3698e+06 | 9.16323e+06 | 3.54325e+07 |
| 51166 | 20221013.3 | 11012.9 | 0 | 40320.4 | 0 | 10266.9 | 0 | 4293.88 | 0 | 3468.82 | 0 | 3209.83 | 0 | 21994 | 6307.98 | 8.32349e+07 | 19594.2 | 1.71746e+07 | 5599.85 | 1.53396e+07 | 2420.73 | 9.57242e+06 | 2029.2 | 1.08831e+07 | 1871.34 | 9.31027e+06 | 889265 | 1.36769e+06 | 9.20851e+06 | 3.54325e+07 |
| 51192 | 20221013.15 | 11236.4 | 0 | 41714.3 | 0 | 10479.4 | 0 | 4385.86 | 0 | 3563.95 | 0 | 3339.92 | 0 | 23541.9 | 6302.95 | 8.24484e+07 | 19661.9 | 1.69124e+07 | 5600.18 | 1.53396e+07 | 2431.32 | 9.57242e+06 | 2033.47 | 9.57242e+06 | 1878.69 | 9.04813e+06 | 902875 | 1.38135e+06 | 9.27532e+06 | 3.58042e+07 |
| 51218 | 20221014.2 | 10047.5 | 0 | 39098.4 | 0 | 10049.4 | 0 | 4310.54 | 0 | 3457.64 | 0 | 3097.68 | 0 | 22728.6 | 6405.26 | 8.55942e+07 | 19215.6 | 1.74367e+07 | 5690.56 | 1.53396e+07 | 2424.67 | 9.83456e+06 | 2036.1 | 9.31027e+06 | 1879.17 | 9.04813e+06 | 892491 | 1.37838e+06 | 9.07399e+06 | 3.55556e+07 |
| 51225 | 20221014.5 | 10867.1 | 0 | 39581.7 | 0 | 10231 | 0 | 4666.19 | 0 | 3424.22 | 0 | 3231.91 | 0 | 29800.6 | 6372.16 | 8.32349e+07 | 20080.7 | 1.66503e+07 | 5542.74 | 1.56017e+07 | 2420.92 | 9.83456e+06 | 2035.03 | 9.31027e+06 | 1867.36 | 9.31027e+06 | 917146 | 1.37633e+06 | 9.13466e+06 | 3.53707e+07 |
| 51268 | 20221014.24 | 10451.7 | 0 | 39975 | 0 | 10156.6 | 0 | 4589.1 | 0 | 3598.76 | 0 | 3219.98 | 0 | 26259.1 | 6175.05 | 8.29727e+07 | 19398.9 | 1.69124e+07 | 5576.74 | 1.53396e+07 | 2422.77 | 9.57242e+06 | 2057.44 | 9.57242e+06 | 1870.29 | 9.04813e+06 | 900135 | 1.40408e+06 | 9.39445e+06 | 3.65714e+07 |
| 51312 | 20221014.44 | 10700 | 0 | 40972.9 | 0 | 10237.2 | 0 | 4689.8 | 0 | 3562.02 | 0 | 3414.58 | 0 | 25408.5 | 6144.6 | 8.32349e+07 | 19380 | 1.69124e+07 | 5583.26 | 1.53396e+07 | 2409.54 | 9.57242e+06 | 2026.31 | 9.31027e+06 | 1865.88 | 9.04813e+06 | 891980 | 1.37652e+06 | 9.26688e+06 | 3.44781e+07 |
| 51323 | 20221017.3 | 11697.2 | 0 | 43984.1 | 0 | 10061.3 | 0 | 4390.77 | 0 | 3444.48 | 0 | 3280.7 | 0 | 23812.4 | 6253.13 | 8.3497e+07 | 19369.6 | 1.6126e+07 | 5600.96 | 1.56017e+07 | 2434.56 | 9.57242e+06 | 2039.86 | 9.31027e+06 | 1907.97 | 9.04813e+06 | 891786 | 1.36989e+06 | 9.20028e+06 | 3.58042e+07 |
app
| build_id | build_number | tpcc_virtual_cft^ | tpcc_virtual_cft_mem | ls_virtual_cft^ | ls_virtual_cft_mem | ls_jwt_virtual_cft^ | ls_jwt_virtual_cft_mem | ls_js_virtual_cft^ | ls_js_virtual_cft_mem | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | ls_full_js_virtual_cft^ | ls_full_js_virtual_cft_mem | ls_js_jwt_virtual_cft^ | ls_js_jwt_virtual_cft_mem | ls_sgx_cft^ | ls_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | hist_sgx_cft^ | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 51264 | 20221014.22 | 11221.2 | 0 | 41594.1 | 0 | 10578 | 0 | 4340.23 | 0 | 6187.49 | 8.27106e+07 | 3553.48 | 0 | 3296.31 | 0 | 19467.1 | 1.69124e+07 | 5549.84 | 1.56017e+07 | 2445.21 | 9.57242e+06 | 26171.7 | 1956.33 | 9.83456e+06 | 1866.56 | 9.04813e+06 | 904833 | 1.37275e+06 | 9.19623e+06 | 3.56174e+07 |
| 51280 | 20221014.29 | 10943.2 | 0 | 40133.3 | 0 | 10784.8 | 0 | 4253.41 | 0 | 6392.17 | 8.32349e+07 | 3623.97 | 0 | 3373.8 | 0 | 19710.7 | 1.71746e+07 | 5560.44 | 1.53396e+07 | 2417.16 | 9.57242e+06 | 24389.8 | 2111.23 | 9.31027e+06 | 1873.28 | 9.31027e+06 | 885848 | 1.36242e+06 | 9.21688e+06 | 3.51286e+07 |
| 51328 | 20221017.5 | 11523.6 | 0 | 41955.5 | 0 | 10464.6 | 0 | 4348.88 | 0 | 6275.97 | 8.32349e+07 | 3692.78 | 0 | 3239.75 | 0 | 19428.5 | 1.6126e+07 | 5574.45 | 1.50774e+07 | 2414.99 | 9.57242e+06 | 31192.4 | 2028.51 | 9.31027e+06 | 1874.3 | 9.04813e+06 | 888384 | 1.36944e+06 | 9.27112e+06 | 3.53713e+07 |
@eddyashton @jumaffre So the daily passed with this change. Is it reasonable to assume this as an indicator to merge or am I missing something else?
@eddyashton @jumaffre So the daily passed with this change. Is it reasonable to assume this as an indicator to merge or am I missing something else?
I realise that this branch was behind main when this first ran so let's make sure it runs now that's updated.
@eddyashton @jumaffre So the daily passed with this change. Is it reasonable to assume this as an indicator to merge or am I missing something else?
If the daily runs and passes, then it's safe to merge.
The only thing this doesn't cover is that some of the tests in the lts_compatibility job are still being skipped because of the OpenAPI prefix change:
https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=51265&view=logs&j=5435e0ac-25e5-5426-50be-61b0d0ea8d34&t=a2a8c6c1-1b13-59a7-596c-97d4f22a6857&l=40608
Test requirements not met, skipping "test_random_receipts": Missing required methods: /log/private
The test requirement decorator runs correctly, but believes its found a service that doesn't support this endpoint. I think we want to ensure this test runs, either with a new decorator requirement ("require either this method OR this method"), or just disabling the requirements checks for lts_compatibility.
@eddyashton I agree, and until there's a way for lts_compatibility test to run safely, it looks like the OpenAPI schema changes for /app will have to revert. I will push a change for it.