Iterations and Games Played diverged on an SPSA

[Viren6]

Some of the iterations are missing somehow:

https://tests.stockfishchess.org/tests/view/66410a7b32d968041c323379

[vondele]

Maybe related to these server logs (a few older ones present, so it is not recent):

sudo journalctl -u fishtest@6543 --since "90 days ago" | grep "incompatible stats" | head
Feb 18 19:17:37 tests.stockfishchess.org pserve[18998]: Update_task: task 65d19ab61d8e83c78bfd8436/193 has incompatible stats. Before 12. Now 12. SPSA_games 12.
Feb 18 19:23:28 tests.stockfishchess.org pserve[18998]: Update_task: task 65d19ab61d8e83c78bfd8436/202 has incompatible stats. Before 16. Now 16. SPSA_games 16.
Feb 18 20:21:15 tests.stockfishchess.org pserve[18998]: Update_task: task 65d10a311d8e83c78bfd784c/143 has incompatible stats. Before 160. Now 128. SPSA_games 0.
Feb 19 23:51:28 tests.stockfishchess.org pserve[18998]: Update_task: task 65d209371d8e83c78bfd8c83/190 has incompatible stats. Before 128. Now 64. SPSA_games 0.
Feb 20 17:58:30 tests.stockfishchess.org pserve[18998]: Update_task: task 65d45d6a1d8e83c78bfdb6a7/70 has incompatible stats. Before 384. Now 352. SPSA_games 0.
Feb 20 17:58:32 tests.stockfishchess.org pserve[18998]: Update_task: task 65d11b811d8e83c78bfd7a5b/391 has incompatible stats. Before 384. Now 352. SPSA_games 0.
Feb 21 22:51:48 tests.stockfishchess.org pserve[18998]: Update_task: task 65d4f8621d8e83c78bfdc16e/118 has incompatible stats. Before 36. Now 36. SPSA_games 12.
Feb 23 19:17:44 tests.stockfishchess.org pserve[3504]: Update_task: task 65d7d7d39b2da0226a5a205b/208 has incompatible stats. Before 70. Now 68. SPSA_games 0.
Feb 29 04:57:55 tests.stockfishchess.org pserve[10538]: Update_task: task 65dcef79347819d80f59b2a9/148 has incompatible stats. Before 132. Now 132. SPSA_games 132.
Feb 29 13:29:44 tests.stockfishchess.org pserve[10538]: Update_task: task 65dcef79347819d80f59b2a9/210 has incompatible stats. Before 72. Now 72. SPSA_games 72.

[vondele]

Some more log history:

Jun 01 19:25:27 tests.stockfishchess.org systemd[1]: Started Fishtest Server port 6543.
Jun 01 19:25:29 tests.stockfishchess.org pserve[25854]: Starting server in PID 25854.
Jun 01 19:25:31 tests.stockfishchess.org pserve[25854]: Request_task: no useful cached runs left
Jun 01 19:26:14 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b642e11645bd3d3fac49c, task_id: 553, worker: technologov-48cores-r114-98ab, reason: 'Finished match uncleanly'
Jun 01 19:27:28 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665a7eea062b2c3cf814fe40, task_id: 4719, worker: vdv-7cores-f3316e2d-e540*, reason: 'Stale active task'
Jun 01 19:28:29 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b642e11645bd3d3fac49c...
Jun 01 19:28:58 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b1965586058766677cf97, task_id: 1030, worker: vdv-7cores-59b42508-775d*, reason: 'Stale active task'
Jun 01 19:31:32 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b730a11645bd3d3fac53b...
Jun 01 19:34:31 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b6b8911645bd3d3fac4d8...
Jun 01 19:34:57 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665a7eea062b2c3cf814fe40, task_id: 4569, worker: vdv-7cores-16769c17-a307*, reason: 'Stale active task'
Jun 01 19:36:22 tests.stockfishchess.org pserve[25854]: dead_tasks_count=1
Jun 01 19:36:22 tests.stockfishchess.org pserve[25854]: dead task: run: https://tests.stockfishchess.org/tests/view/665b730a11645bd3d3fac53b task_id: 461 worker: vdv-7cores-edd0379f-6348*
Jun 01 19:37:24 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b1965586058766677cf97...
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 229, worker: vdv-7cores-b9e6610b-faba*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 202, worker: vdv-7cores-cfd07497-0d09*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 164, worker: vdv-7cores-25c3667c-8e84*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 182, worker: vdv-7cores-82a978ef-5320*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 161, worker: vdv-7cores-1c2466a7-ba6c*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:38:46 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 171, worker: vdv-7cores-44742f32-b5f4*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:11 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 173, worker: vdv-7cores-5394327f-e0be*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b1965586058766677cf97, task_id: 815, worker: vdv-7cores-cb31a66d-5255*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b501c11645bd3d3fac2e9, task_id: 816, worker: vdv-7cores-5278703f-ee36*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b730a11645bd3d3fac53b, task_id: 59, worker: vdv-7cores-e945f0c2-38de*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 270, worker: vdv-7cores-49c9014b-cd79*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 234, worker: vdv-7cores-d9f15f62-4c4f*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:16 tests.stockfishchess.org pserve[25854]: Failed_task: failure for: https://tests.stockfishchess.org/tests/view/665b77666658f2adece78646, task_id: 250, worker: vdv-7cores-fabc3726-3134*, reason: 'Post request to https://tests.stockfishchess.org:443/api/request_spsa failed'
Jun 01 19:39:17 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b1965586058766677cf97/815 is not active
Jun 01 19:39:18 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b501c11645bd3d3fac2e9/816 is not active
Jun 01 19:39:18 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b730a11645bd3d3fac53b/59 is not active
Jun 01 19:39:19 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b77666658f2adece78646/270 is not active
Jun 01 19:39:20 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b77666658f2adece78646/234 is not active
Jun 01 19:39:20 tests.stockfishchess.org pserve[25854]: Request_spsa: task 665b77666658f2adece78646/250 is not active
Jun 01 19:39:46 tests.stockfishchess.org pserve[25854]: Update_task: task 665a8e00062b2c3cf814fe8d/1296 has incompatible stats. Before 252. Now 252. SPSA_games 28.
Jun 01 19:39:46 tests.stockfishchess.org pserve[25854]: Update_task: task 665a8e00062b2c3cf814fe8d/1339 has incompatible stats. Before 252. Now 252. SPSA_games 28.
Jun 01 19:39:46 tests.stockfishchess.org pserve[25854]: Update_task: task 665b501c11645bd3d3fac2e9/885 has incompatible stats. Before 252. Now 252. SPSA_games 28.
Jun 01 19:39:46 tests.stockfishchess.org pserve[25854]: Update_task: task 665ae6320223e235f05b7d14/4588 is not active
Jun 01 19:39:47 tests.stockfishchess.org pserve[25854]: Failed_task: task 665a8e00062b2c3cf814fe8d/1296 is not active
Jun 01 19:39:47 tests.stockfishchess.org pserve[25854]: Failed_task: task 665b501c11645bd3d3fac2e9/885 is not active
Jun 01 19:39:47 tests.stockfishchess.org pserve[25854]: Failed_task: task 665a8e00062b2c3cf814fe8d/1339 is not active
Jun 01 19:40:27 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b78a46658f2adece7864e...
Jun 01 19:43:18 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 6640edea70e85e1f2ce321f9...
Jun 01 19:46:14 tests.stockfishchess.org pserve[25854]: Validate_random_run: validated aggregated data in cache run 665b367ad82a32c29777c010...

looks like these incompatible stats relate directly to failed post requests to request_spsa