Azure
CLI option --poll-until-tasks-complete is too slow on many tasks
Problem Description
I use cli option --poll-until-tasks-complete too wait until all jobs completed. This option is too slow when you have many tasks (e. g. 80.000). So this option is unusable for me.
Batch Shipyard Version
3.9.1
You can also try the counting API: shipyard jobs tasks counts. Note that this is eventually consistent.
If that is insufficient, is it possible to add task deletion into your workflow to reduce the count? Otherwise, there will need to be a feature to support OData filtering on that command.
My use case: I have about 170 jobs with each 500 tasks. Together these are 85.000 tasks (total task number).
Workaround 1: shipyard jobs tasks count:
Here I have first compute the total task number with the command above. Than I have to write a while loop with this command and compare output line * completed: . Are you seeing a better solution here?
Workaround 2: Delete tasks when they are completed:
How could this be easily done? And if this could be done, the first call of poll-until-tasks-complete would be unusable slow as before.
Workaround 3: Do you have other ideas?
Apologies for the delay:
Workaround 1, that's correct except you should be able to programmatically parse the output with --raw.
Workaround 2, correct, a filter will need to be implemented.
For other workarounds, unfortunately you may need to perform a short term integration with Azure CLI and combine some features between the two.
What about the stats command?
shipyard jobs stats
Could this be fast enough? Does this only output jobs specified in the jobs configuration file or does this command lists all jobs in the batch account? The documentation is not clear to me:
"stats will generate a statistics summary of a job or jobs"
Apologies for the delay. Without a --jobid it will list all jobs in the account. Internally it will call get task counts for each job which is the same as Workaround 1.