503 The service is currently unavailable error during the Google AutoML warmup process

[mathjazz]

We're hitting lots of "503 The service is currently unavailable" errors in the Google AutoML warmup process.

With the increase in the number of AutoML locales the time to do the actual warmup also increases. We are at ~20 locales and it almost always takes more than the warmup job frequency (1 minute) to warm them all up.

Since we can only run one task in parallel, we no longer warmup locales every minute. The longer the time between the warmups, the higher the chance of hitting the cold engine, manifested as "503 The service is currently unavailable".

We use APScheduler for that task, so the simplest trick to test is to increase the number of instances that can run at the same time.