Azure Fusion integration should not fail when managed identity auth is present

[swampie]

New feature

Usage scenario

Now that Azure Managed Identity has been merged the support for fusion should also be enabled. The goal of this feature is to prevent failures when a managed identity is correctly configured and pass the configuration to fusion.

Suggest implementation

• one change should be the removal/extension of this logic

[swampie]

one possibility to prevent changes on fusion (if it relies on sas tokens) is to change the getOrCreateSasToken() to try to generate a token with the managed identity (unsure if this is possible)

[alberto-miranda]

I'm currently writing a PR solving this in order to unblock https://github.com/seqeralabs/fusion/issues/526, but there's one thing I don't understand: why are we always generating a SAS token with getOrCreateSasToken() and returning it as a result.AZURE_STORAGE_SAS_TOKEN even if the accountKey authentication method has been configured?

I was initially thinking about setting result.AZURE_STORAGE_SAS_TOKEN only for if the SAS token auth was requested, but I don't want to break any existing logic relying on this.

[adamrtalbot]

returning it as a result.AZURE_STORAGE_SAS_TOKEN even if the accountKey authentication method has been configured?

I believe the SAS key is created by Nextflow, which is sent to the worker nodes as a temporary token for Fusion to use. This is because the accountKey isn't available to worker nodes (?). We can simply bypass this if we believe the worker nodes have an MI associated.

Relates to https://github.com/nextflow-io/nextflow/issues/5232 which is the non-Fusion equivalent.

[alberto-miranda]

returning it as a result.AZURE_STORAGE_SAS_TOKEN even if the accountKey authentication method has been configured?

I believe the SAS key is created by Nextflow, which is sent to the worker nodes as a temporary token for Fusion to use. This is because the accountKey isn't available to worker nodes (?). We can simply bypass this if we believe the worker nodes have an MI associated.

Relates to https://github.com/nextflow-io/nextflow/issues/5232 which is the non-Fusion equivalent.

The thing is that AFAICT, Fusion only uses the SAS token if Nextflow has not been configured with an accountKey (cc @pabloaledo), which is why I don't get why we are always generating one here (no matter what auth is used). I can maintain the current logic, but I feel it's a remnant and probably not required nowadays.

[alberto-miranda]

returning it as a result.AZURE_STORAGE_SAS_TOKEN even if the accountKey authentication method has been configured?

I believe the SAS key is created by Nextflow, which is sent to the worker nodes as a temporary token for Fusion to use. This is because the accountKey isn't available to worker nodes (?). We can simply bypass this if we believe the worker nodes have an MI associated. Relates to #5232 which is the non-Fusion equivalent.

The thing is that AFAICT, Fusion only uses the SAS token if Nextflow has not been configured with an accountKey (cc @pabloaledo), which is why I don't get why we are always generating one here (no matter what auth is used). I can maintain the current logic, but I feel it's a remnant and probably not required nowadays.

We actually got a reply to this with #5286 :sob:, the SAS token is required by Fusion even if SAS token authentication has not been explicitly requested in the nextflow configuration.

[adamrtalbot]

We actually got a reply to this with https://github.com/nextflow-io/nextflow/issues/5286 😭, the SAS token is required by Fusion even if SAS token authentication has not been explicitly requested in the nextflow configuration.

We assume that if Nextflow is using a Managed Identity, Fusion can also use the same identity. A future option could be a config item saying "Do the workers have a managed identity, if so what is the ID?" This could connect with issue https://github.com/nextflow-io/nextflow/issues/5232 which is the non-Fusion version.

On AWS we have the following settings, we could have a similar format for Azure:

aws.batch.jobRole = 'JOB ROLE ARN'
aws.batch.executionRole = 'EXECUTION ROLE ARN'

e.g.:

azure.batch.managedIdentity = 'MANAGED_IDENTITY_CLIENT_ID'

or even a per-pool setting:

azure.batch.pools.<pool name>.managedIdentity = 'MANAGED_IDENTITY_CLIENT_ID'

When not using a Managed Identity, Nextflow will generate a SAS key using it's own authentication which it passes to Fusion (for now!). This is very simple and robust, but it may be disabled by very security conscious users.

For now, Fusion must use the following settings (generally configured by Nextflow):

• if there is a managed ID, we assume all is the same and use it
• if there is not a managed ID, but there is a key or service principle, generate a SAS and use it
• if there is a SAS, use it directly
• if none, throw an error before submitting a process