data icon indicating copy to clipboard operation
data copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

Update LAION-example and default value of `scheduled_tasks`

Open SvenDS9 opened this issue 1 year ago • 1 comments

Changes

  • Update LAION-example to use threadpool_map
  • Change default value of scheduled_tasks to 500

SvenDS9 avatar Mar 16 '23 09:03 SvenDS9

128 was still to low - at least with a timeout of 5 seconds. With this new value I am mostly limited by my internet connection.

I have also compared the speed to async_map. With the correct values (high batch_size e.g. 800 and high enough max_concurrency e.g. 64) I get similar performance - also limited by my internet connection.

SvenDS9 avatar Mar 16 '23 09:03 SvenDS9