Maximum number of running tasks allowed in batch mode

[psytky03]

I submitted a job which contains 5,000 tasks with dsub. It seems tasks are processed in sequential manner, and only 7-8 tasks are processed at the same time and some are under pending status with a message "A resource limit has delaye..".

Is there any way I can increase the resources which allow a faster data process?

[mbookman]

Hi @psytky03 .

The number of concurrent jobs you can run is going to be gated by your Compute Engine Quota. Default quotas are typically rather low to protect new Cloud users from accidental over-spend.

Google Genomics Pipelines operations are typically throttled by one of the following quotas:

• CPUs
• Persistent Disk
• IP addresses

You can get more details on which quotas are limiting by either running dstat ... --full to see more detail for specific jobs or to get a higher-level view of things, follow the instructions at Checking your quota.

Once you have observed which quotas are limiting, then follow the instructions at Requesting an increase in quota.

[psytky03]

Thanks for explaining the system. I will check and decide an appropriate quota level.

[mbookman]

Leaving this open as a doc bug: add some documentation around GCP quotas.