Update TES executor to TES API v1.1

[bentsherman]

Summary of changes:

• Update TES Java SDK to TES 1.1 spec
• Deprecate NXF_EXECUTOR_TES_ENDPOINT in favor of tes.endpoint config option
• Add support for output globs
• Add support for output directories
• Update docs
• Add README with instructions for Java SDK generation

Unfortunately Funnel hasn't been updated to 1.1 yet, so I don't have an easy way to test it at the moment.

@vsmalladi can you test this PR with Azure?

git clone https://github.com/nextflow-io/nextflow -b tes-update-1.1
cd nextflow
make compile

./launch.sh run ...

[netlify[bot]]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	0b86d0f9e88ceabf67e5d72255b29d7c016834db
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/663e0815e503ae00080f80b5
😎 Deploy Preview	https://deploy-preview-4195--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[kellrott]

BTW, v1.1 support is being tested in Funnel PR 716

[bentsherman]

Release candidate for Funnel with TES 1.1 support: https://github.com/ohsu-comp-bio/funnel/releases/tag/0.11.0-rc.2

[lbeckman314]

Hi @bentsherman! Just wanted to reach out and check if there were any updates or blockers with this PR that we can help with. Let us know and we'll happy to take a look!

[bentsherman]

We're just trying to get TES Azure to work and then I think we'll merge

[vsmalladi]

Have an outstanding PR that is almost done. Should be able to finish this week.

[vsmalladi]

@bentsherman Whats the next steps to get this out of draft?

[bentsherman]

@vsmalladi I just looked over everything in total and it looks good, you should probably do another test with TES Azure to make sure it still works with my last round of edits

[vsmalladi]

@bentsherman built and hit this error:

ERROR ~ Unexpected error [ProviderMismatchException]

Thoughts?

[bentsherman]

Need more context that just the error message. How you ran it, the stack trace, etc

[vsmalladi]

@bentsherman test pass. I am able to submit to azure now. using this build.

I think we can merge this in.

[lbeckman314]

Overview

Tests passing with latest Nextflow PR and Funnel. Good to merge! :white_check_mark:

Installing Nextflow PR 🐍

➜ gh pr checkout 4195
# Switched to branch 'tes-update-1.1'

➜ make compile
# ./gradlew compile exportClasspath
# Starting a Gradle Daemon (subsequent builds will be faster)
# 
# BUILD SUCCESSFUL in 19s
# 49 actionable tasks: 37 executed, 12 from cache
# DONE Thu May  9 18:55:00 PDT 2024

Starting Funnel Server 🌪️

➜ brew install funnel
# ==> Installing funnel from ohsu-comp-bio/formula
# 🍺  /opt/homebrew/Cellar/funnel/0.11.0: 5 files, 81.8MB, built in 4 seconds

➜ which funnel
# /opt/homebrew/bin/funnel

➜ funnel version
# git commit: a514ca2b
# git branch: master
# git upstream: [email protected]:ohsu-comp-bio/funnel.git
# build date: 2024-04-27T04:02:37Z
# version: 0.11.0

➜ funnel server run
# server               Server listening
# httpPort             8000
# rpcAddress           :9090
# time                 2024-05-09T19:03:48-07:00

Running nf-canary tests ✅

➜ cd ~/nf-canary

➜ cat <<EOF >> nextflow.config
// Config to use TES Executor
plugins {
  id 'nf-ga4gh'
}
process.executor = 'tes'
tes.endpoint = 'http://localhost:8000'
EOF

➜ ~/nextflow/launch.sh run main.nf

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [fabulous_lamarr] DSL2 - revision: d61ea81ba6

# Uploading local `bin` scripts folder to /Users/beckmanl/nf-canary/work/tmp/3b/7c490ef46b32227c138d65b5662de4/bin
# executor >  tes [http://localhost:8000] (13)
# executor >  tes [http://localhost:8000] (14)
# executor >  tes [http://localhost:8000] (14)
# executor >  tes [http://localhost:8000] (14)
# executor >  tes [http://localhost:8000] (14)
# executor >  tes [http://localhost:8000] (14)
[9d/473866] NF_CANARY:TEST_SUCCESS            | 1 of 1 ✔
[9f/583360] NF_CANARY:TEST_CREATE_FILE        | 1 of 1 ✔
[75/e6598e] NF_CANARY:TEST_CREATE_EMPTY_FILE  | 1 of 1 ✔
[60/75354d] NF_CANARY:TEST_CREATE_FOLDER      | 1 of 1 ✔
[7b/248ba5] NF_CANARY:TEST_INPUT (1)          | 1 of 1 ✔
[0c/bf0a94] NF_CANARY:TEST_BIN_SCRIPT         | 1 of 1 ✔
[-        ] NF_CANARY:TEST_STAGE_REMOTE       -
[3f/383a73] NF_CANARY:TEST_PASS_FILE          | 1 of 1 ✔
[51/d0335c] NF_CANARY:TEST_PASS_FOLDER        | 1 of 1 ✔
[91/e8ad57] NF_CANARY:TEST_PUBLISH_FILE       | 1 of 1 ✔
[8a/58f714] NF_CANARY:TEST_PUBLISH_FOLDER     | 1 of 1 ✔                
[ea/19b2cb] NF_CANARY:TEST_IGNORED_FAIL       | 1 of 1, failed: 1 ✔                  
[f3/99efe0] NF_CANARY:TEST_MV_FILE            | 1 of 1 ✔                             
[ce/f5fcce] NF_CANARY:TEST_MV_FOLDER_CONTENTS | 1 of 1 ✔                             
[ae/8d5187] NF_CANARY:TEST_VAL_INPUT          | 1 of 1 ✔                            
[ea/19b2cb] NOTE: Process `NF_CANARY:TEST_IGNORED_FAIL` terminated with an error exit status (1) -- Error is ignored