improve container credentials retrieval using workflow compute environment

[munishchouhan]

1. Added workflowId in PlatformId
2. Added Models to get workflow information
3. Added API call in TowerClient to get workflow information using workflowId
4. Added findComputeCreds to get the credential id from workflow information
5. Added Unit Test

[munishchouhan]

@pditommaso how can I test this feature locally?

[pditommaso]

1. run tower in your computer
2. run wave in your compute
3. create AWS Batch Compute env having permissions to pull containers from AWS ECR
4. run nextflow script using a container in that ECR using local tower and wave

[munishchouhan]

@pditommaso I have tried using Tower also, but same error Can you please share, where nextflow sends workflow id to wave?

[munishchouhan]

ok i found its there in wave plugin https://github.com/nextflow-io/nextflow/blob/f0d5cc5c674e4e18424e431a5f50930dcd9906c2/plugins/nf-wave/src/main/io/seqera/wave/plugin/SubmitContainerTokenRequest.groovy#L57

[munishchouhan]

@pabloaledo I have used the master branch of nextflow but still got the same error

[munishchouhan]

@pditommaso SubmitContainerTokenRequest in Tower doesn't have WorkflowId in it should i add it there? https://github.com/seqeralabs/nf-tower-cloud/blob/14a5582fe2aa84b5fda9706e3f6776d9084ced8c/tower-enterprise/src/main/groovy/io/seqera/wave/SubmitContainerTokenRequest.groovy#L32-L79

[pditommaso]

it should *not* be the problem. The request is made by Nextflow (that's not used by tower right now)

[munishchouhan]

ok i will make the changes in tower

[pditommaso]

My bad, I meant "It should not be the problem"

[pditommaso]

The flow is like this

• Tower adds TOWER_WORKFLOW_ID in the launcher job
• The tower client in Nextflow fetchs it and include in the request sent to wave
• the rest continues in Wave

Make sure the Wave client find it and submit correctly

[munishchouhan]

Found the issue The parameter name was different in Wave-API and Nextflow should we use wave-api in nextflow and tower for wave models so that it remains consistent?

Created new PR for Wave-API https://github.com/seqeralabs/libseqera/pull/6

[marcodelapierre]

Munish: tests are passing locally

@pditommaso you can go ahead and review then

[marcodelapierre]

@pditommaso heads up that this requires a new Wave API release

[munishchouhan]

I have tested using local compute, got this INFO i.s.w.service.CredentialServiceImpl - ------Found compute env: local-platform and credentials: null I will test with aws-batch next

[munishchouhan]

pipeline with aws-batch is not running properly in local I will try some alternative

[munishchouhan]

I have tested this PR with local-platform after correcting the credentialsId type for sanity check I got this, which

16:10:21.174 [io-executor-thread-2] INFO  i.s.w.service.CredentialServiceImpl - Platform = local-platform and credentials.id = 6vmM91lfpEu0E7RGP5Ab9a
16:10:21.174 [io-executor-thread-2] DEBUG i.s.w.service.CredentialServiceImpl - Credentials matching criteria registryName=195996028523.dkr.ecr.eu-west-1.amazonaws.com; userId=1; workspaceId=null; endpoint=http://localhost:8008/api => CredentialsDescription(id:6vmM91lfpEu0E7RGP5Ab9a, provider:null, registry:null)

I am not able to test locally with AWS-BATCH. I have tried the same pipeline seqera cloud and got this error, while creating compute

[munishchouhan]

@pditommaso I tried testing this PR in local with AWS Batch, but it doesn't work can we test this in stage or dev?

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
8498070	Triggered	Username Password	93bb70070c05c68bc280c457d5a41abb46e7be60	src/test/groovy/io/seqera/wave/auth/RegistryCredentialsProviderTest.groovy	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[pditommaso]

@munishchouhan Any chance to fix failing tests at your convenience ?

[munishchouhan]

@munishchouhan Any chance to fix failing tests at your convenience ?

sure

[munishchouhan]

Tested Blob Transfer: Successful

 % wave -i cr.seqera.io/public/nf-jdk:corretto-17-al2023-jemalloc --wave-endpoint https://wave.stage-seqera.io
wave.stage-seqera.io/wt/XXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
(base) munish.chouhan@Munishs-MacBook-Pro ~ % docker pull wave.stage-seqera.io/wt/XXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
corretto-17-al2023-jemalloc: Pulling from wt/XXXXXX/public/nf-jdk
860904071dc6: Pull complete
82160a56be4d: Pull complete
d2d64551932e: Pull complete
f7d1bc77ad09: Pull complete
87ca65aa7e06: Pull complete
Digest: sha256:3f9cf279c1ad0454244469eb52f955dc41072465b310f238e119cd8cebb6f067
Status: Downloaded newer image for wave.stage-seqera.io/wt/XXXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
wave.stage-seqera.io/wt/XXXXXX/public/nf-jdk:corretto-17-al2023-jemalloc

[munishchouhan]

Tested build, scan and build log transfer: successful

% /bin/zsh /Users/munish.chouhan/testing_ground/wave_testing/build-images_stage.sh
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard--26c87e08d44802ba
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:salmon_numpy--31f71aba34cc9f18
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
picard_numpy--e90fce5ae5a4f7c7: Pulling from wt/xxxxxxx/wave/build/stage
6360b3717211: Pull complete 
2ec3f7ad9b3c: Pull complete 
7716ca300600: Pull complete 
4f4fb700ef54: Pull complete 
8c61d418774c: Pull complete 
03dae77ff45c: Pull complete 
aab7f787139d: Pull complete 
837d55536720: Pull complete 
897362c12ca7: Pull complete 
3893cbe24e91: Pull complete 
d1b61e94977b: Pull complete 
57d9b5e475d4: Pull complete 
a4c883d12ac5: Pull complete 
Digest: sha256:371f85c396a177dfa243323cdcdd63273b4589a7b9e5252bd45bbdefdd130470
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview wave.stage-seqera.io/wt/f7d4cfa17e01/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard--26c87e08d44802ba
picard--26c87e08d44802ba: Pulling from wt/xxxxxxx/wave/build/stage
6360b3717211: Already exists 
2ec3f7ad9b3c: Already exists 
7716ca300600: Already exists 
4f4fb700ef54: Already exists 
8c61d418774c: Already exists 
03dae77ff45c: Already exists 
aab7f787139d: Already exists 
837d55536720: Already exists 
897362c12ca7: Already exists 
3893cbe24e91: Already exists 
d1b61e94977b: Already exists 
b630d4f4ff7e: Pull complete 
744b79fad90d: Downloading [=============>                                     ]  184.4MB/674.6MB

[munishchouhan]

Test - Container pull: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -i ubuntu --wave-endpoint https://wave.stage-seqera.io
wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
latest: Pulling from wt/xxxxxxxx/library/ubuntu
eed1663d2238: Pull complete 
Digest: sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest

Test - Build using dockerfile: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -f Dockerfile --wave-endpoint https://wave.stage-seqera.io --tower-token xxxxxxxx  --tower-endpoint https://api.cloud.stage-seqera.io
wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
b4347a6d3486b02c: Pulling from wt/xxxxxxxx/wave/build/stage
ec562eabd705: Pull complete 
Digest: sha256:e9119a211bb40231e677fad0f3f60d2c3bb94fcdcb3b752330a908107889cd55
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c

[munishchouhan]

Test: build singularity, freeze and user build repository: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -s --conda-package pandas --freeze --build-repo docker.io/hrma017/dev  --wave-endpoint https://wave.stage-seqera.io --tower-token xxxxxxx  --tower-endpoint https://api.cloud.stage-seqera.io --platform linux/arm64
oras://docker.io/hrma017/dev:pandas--e7eed9f3222a1997
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing %  docker run  --privileged quay.io/singularity/singularity:v3.11.4-slim-arm64 pull oras://docker.io/hrma017/dev:pandas--e7eed9f3222a1997
INFO:    Downloading oras image

[pditommaso]

@munishchouhan all green in your side?

[munishchouhan]

@munishchouhan all green in your side?

testing one last thing. will update here soon

[munishchouhan]

test: ran rnasef to create fusion-based images: successful

(base) munish.chouhan@Munishs-MacBook-Pro example-bonus % bash run.sh
Nextflow 24.04.2 is available - Please consider updating your version to it
N E X T F L O W  ~  version 23.10.1
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [55133f624d]
Launching `https://github.com/nextflow-io/rnaseq-nf` [happy_volhard] DSL2 - revision: 88b8ef803a [master]
 R N A S E Q - N F   P I P E L I N E
 ===================================
 transcriptome: /Users/munish.chouhan/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
 reads        : /Users/munish.chouhan/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_gut_{1,2}.fq
 outdir       : results
 
executor >  local (fusion enabled) (4)
[66/01293a] process > RNASEQ:INDEX (ggal_1_48850000_49020000) [100%] 1 of 1 ✔
[51/3d1445] process > RNASEQ:FASTQC (FASTQC on ggal_gut)      [100%] 1 of 1 ✔
[2a/8846cc] process > RNASEQ:QUANT (ggal_gut)                 [100%] 1 of 1 ✔
[ed/f7a053] process > MULTIQC                                 [100%] 1 of 1 ✔

Done! Open the following report in your browser --> results/multiqc_report.html

Completed at: 09-Jul-2024 17:23:06
Duration    : 2m 10s
CPU hours   : 0.1
Succeeded   : 4

[munishchouhan]

@pditommaso All green from my side