Improve container credentials retrieval using workflow compute environment

[pditommaso]

Wave retrieves the container credentials to be used, querying Tower credentials for container-reg provider and looking for a matching registry name.

This is convenient for users that rely on registries such as Docker or Quay. However, it can be suboptimal for users that store container images in the cloud registry provided by the cloud vendor associated with by the pipeline execution.

The goal of this issue is to extend the credentials retrieval mechanism in such a way that when there isn't a credentials match for the required repositories, the system fallbacks to the crendianls associated with the compute environment used by the pipeline execution.

The above approach should take place only when the container repository is hosted in the same platform where the pipeline executed e.g. AWS ECR and the execution platform is AWS Batch

[pditommaso]

Implementation notes

This feature requires making Wave aware of the ID of the workflow that requires the access to a specific container, therefore:

1. Add the workflowId attribute in SubmitContainerTokenRequest
2. The workflow id is included in the ContainerRequestData
3. When looking for the container repository credentials and no match is found: a. the workflow id is used to lookup the credentialsId and platform the corresponding launch using the Tower endpoint /workflow/{workflowId}/launch b. the credentialsId is used to retried the corresponding platform key in a similar manner as it's done for the container credentials here

Caveats

• This feature can only be implemented for workflow running on AWS Batch (platform == 'aws-batch') or Google Batch (platform == 'google-batch') CEs
• Currently Wave is only retrieving Credentials having provider == container-reg
• This feature requires to requires to retrieve the credentials keys for the provider == aws and google
• Those keys should be mapped into the corresponding ContainerRegistryKeys structure

[munishchouhan]

@pditommaso do we have any tower API where i can get response type of /workflow/{workflowId}/launch or do i need to recreate it in wave?

[pditommaso]

Do you mean API client? yes, see here

https://github.com/seqeralabs/wave/blob/82682b63b450de654e0cb033c35270562df71b59/src/main/groovy/io/seqera/wave/tower/client/TowerClient.groovy#L60-L82

[munishchouhan]

I meant Classes to deserialize response from [/workflow/{workflowId}/launch] there are a lot of classes that need to be created for the response and we only need credentialsId and platform

[pditommaso]

We don't have spared client classes. I think it's not needed to replicate the overall model, it should be enough the DescribeWorkflowLaunchResponse and only the required fields in the nested objects. See here

https://github.com/seqeralabs/nf-tower-cloud/blob/5299c433adb3aeb50a05fc1cee24f62062e75d3c/tower-enterprise/src/main/groovy/io/seqera/tower/exchange/workflow/DescribeWorkflowLaunchResponse.groovy#L21-L31

[robnewman]

@munishchouhan What is the status of this issue? Thanks.

[canny[bot]]

This issue has been linked to a Canny post: Authenticate to ECR using AWS roles :tada:

[munishchouhan]

@munishchouhan What is the status of this issue? Thanks.

Dev part is completed, but I have to test it on dev because not possible to test in local Testing of this feature requires some special setup, where credentials can be used to create compute and access ECR I tried multiple times to test in dev using tower dev user but no luck so to summarise its testing is pending, once that done paolo can review it