Investigate potential autoscaledpool deadlock when client_info=1

[Pijukatel]

Problem:

client_info=1 of autoscaled pool can completely stop the crawler, without any recovery. Investigate and fix if not desired.

Context:

A benchmark test was running for Parsel crawler. Crawler was set up in a way that it was scraping one site hosted locally by server. This eliminated any network delays during scraping as all the server responses were nearly instant. Requests to API were kept as it is, so for example request queue requests were still traveling over network to the real Apify API.

https://console.apify.com/actors/UdJ0NQp2j8140G9db/runs/ZPlTj1VgMLQA7Yuxl#log

After a while the crawler got stopped forever by client_info = 1.0

... 2025-05-21T08:44:09.332Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6583 ... 2025-05-21T08:44:09.333Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6581 ... 2025-05-21T08:44:09.339Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6582 ... 2025-05-21T08:44:57.719Z [ParselCrawler] INFO Current request statistics: 2025-05-21T08:44:57.720Z ┌───────────────────────────────┬─────────────┐ 2025-05-21T08:44:57.721Z │ requests_finished │ 3736 │ 2025-05-21T08:44:57.722Z │ requests_failed │ 0 │ 2025-05-21T08:44:57.723Z │ retry_histogram │ [3736] │ 2025-05-21T08:44:57.724Z │ request_avg_failed_duration │ None │ 2025-05-21T08:44:57.725Z │ request_avg_finished_duration │ 0.269101 │ 2025-05-21T08:44:57.725Z │ requests_finished_per_minute │ 1245 │ 2025-05-21T08:44:57.726Z │ requests_failed_per_minute │ 0 │ 2025-05-21T08:44:57.727Z │ request_total_duration │ 1005.360811 │ 2025-05-21T08:44:57.728Z │ requests_total │ 3736 │ 2025-05-21T08:44:57.728Z │ crawler_runtime │ 180.038534 │ 2025-05-21T08:44:57.729Z └───────────────────────────────┴─────────────┘ 2025-05-21T08:44:57.730Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 14; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 1.0 ...