Support not using a container in Azure Batch

[adamrtalbot]

There are rare cases where a user needs to run a process on a bare VM on Azure Batch. Most notably, when using the Dragen which requires you to use a specific machine image that does not support a Docker container.

Previously, Nextflow prevented this, but this PR removes that safeguard and replaces it with a warning. It's not a good idea, but it can unblock users if they need to use a bare VM.

If nothing else, this PR functions as proof it can be done if anyone else needs to use it.

[netlify[bot]]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	1587653924ff609b5e3a2a994a89475c1b8dc854
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67e58efc644ad800084259e3

[tom-seqera]

It's not a good idea, but it can unblock users if they need to use a bare VM.

Given this, would it be worth keeping the strict validation on by default and instead providing an override config option to turn it off (like azure.requireContainer=false or something) then updating the validation error message to explain that the option exists but you should only use it if you know what you're doing?

[adamrtalbot]

Given this, would it be worth keeping the strict validation on by default and instead providing an override config option to turn it off (like azure.requireContainer=false or something) then updating the validation error message to explain that the option exists but you should only use it if you know what you're doing?

That's a great idea!

[adamrtalbot]

@tom-seqera implemented now as azure.batch.requireContainer.

I'm still not sure this is a wise thing to do, but it unblocks some very specific use cases and it's a relatively buried feature.

[pditommaso]

My understanding is that some tasks my require a container, some other may not, therefore a global setting is sub-optimal.

Also, could not be Dragen accessed by the container anyway? this is for example how it works with AWS Batch (the dragen path in the host is made accessible in the container context)

[adamrtalbot]

Also, could not be Dragen accessed by the container anyway? this is for example how it works with AWS Batch (the dragen path in the host is made accessible in the container context)

Our implementation only works on AWS using a custom AMI. The AMI provided by Illumina uses CentOS 7, which is EOL. And obviously, it's AWS only.

The official AMI provided by Illumina for Azure is not compatible with Docker and they do not want to update it. So we end up either using a Docker compatible image which does not work on Azure Batch, or a specific Dragen based image which does not use Docker. We spent a long time playing around with this and this was the most parsimonious solution we could make work.

[pditommaso]

The official AMI provided by Illumina for Azure is not compatible with Docker

What this means exactly?

[adamrtalbot]

Illumina provide a machine image for Dragen, it’s not compatible with Docker.