apify
`SystemStatus._get_system_info` called too often on same data (optimization candidate)
In AutoscaledPool._worker_task_orchestrator there is a call to self._system_status.get_current_system_info() this processes existing snapshots to calculate whether the system is overloaded or not. This function gets called in while loop without wait and is running as frequently as possible. It involves manipulation of 4 lists of snapshots objects - sorting, and calculating time weighted ratio of overloaded and non overloaded samples.
This is not necessarily useful to be done so frequently as the new snapshot data is usually gathered with some sleep time and thus the snapshot overload calculation is most often running on same inputs again and again.
This is just an optimization candidate to reduce CPU load if needed. This should not be done until sensitive enough benchmark is in place and until the cpu related performance needs to be optimized.
One option to deal with this would be to add some short sleep in _worker_task_orchestrator not call _worker_task_orchestrator while loop so frequently and allow the possibility to add more than one new task at a time if possible, which would also speed up response to reach desired_concurency without spamming calls too frequently.
It would also prevent apify RQ client to spam list_and_head API calls to see if the RQ is empty or not
... 2025-08-06T07:47:54.117Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.144Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.145Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.164Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.184Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.185Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.211Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.232Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.233Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.263Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.950Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.959Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:54.997Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:55.018Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:55.020Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head 2025-08-06T07:47:55.049Z [apify.storage_clients._apify._request_queue_client] INFO self._api_client.list_and_lock_head ...
Example simple way to deal with this: https://github.com/apify/crawlee-python/compare/master...optimize-resource-meauserment