Unexpected response for request (when using nf-shard with a ssh tunnel)

[jfy133]

For a variety of reasons related to the network setup at my institute, I'm currently trying to get nf-shard set up where I have to do a ssh port tunneling to allow my machine running nextflow to contact the VM running nf-shard.

I first have nf-shard running with the docker compose --profile all up command on the VM

Then on my machine I set up the tunnel with ssh -L 10001:<vm>:3000 <intermediate_machine>

While I can now access the nf-shard interface via localhost:10001 and everything works nicely, when I run Nextflow with the following config (actually embedded in a profile called nfshard):

tower {
		enabled = true
		accessToken = "x"
		endpoint = "http://localhost:10001/api"
}

The nextflow run

NXF_VER=23.10.1 nextflow run nf-core/taxprofiler -r 1.1.5 -profile test_nothing,singularity,nfshard --outdir ./results

starts, but then hangs and eventually dies.

The nextflow.log file reports:

Apr-16 10:04:29.750 [main] DEBUG nextflow.util.SimpleHttpClient - Got HTTP error=500 waiting for 750ms (errorCount=1)
Apr-16 10:04:30.557 [main] DEBUG nextflow.util.SimpleHttpClient - Got HTTP error=500 waiting for 2250ms (errorCount=2)
Apr-16 10:04:32.869 [main] DEBUG nextflow.util.SimpleHttpClient - Got HTTP error=500 waiting for 6750ms (errorCount=3)
Apr-16 10:04:39.690 [main] DEBUG nextflow.util.SimpleHttpClient - Got HTTP error=500 waiting for 20250ms (errorCount=4)
Apr-16 10:05:00.033 [main] DEBUG nextflow.util.SimpleHttpClient - Got HTTP error=500 waiting for 60750ms (errorCount=5)
Apr-16 10:06:00.863 [main] DEBUG io.seqera.tower.plugin.TowerClient - Unexpected HTTP response
- endpoint    : http://localhost:10001/api/trace/31_qaJnDV_XfEnq3/begin
- status code : 500
- response msg: {"error":{"name":"PrismaClientValidationError","clientVersion":"5.3.1"}}

My understaning is status 500 is a generic one...

Can you give any suggestions on how to debug this (e.f. with an nf-shard log, if one exists), or is this entirely something to do with my institutes network setup?

[diegomscoelho]

Hi there,

I had same error a few weeks ago, you can "fix it" by downgrading your nextflow to NXF_VER=23.04.4. When I tried using 23.10.1 stopped working as well.

[jfy133]

Hi @diegomscoelho thanks for letting meknow! I'll try it out... but I guess there is some API change which means nf-shard won't work for the itme being I guess :(

[gluonfield]

Hi guys, I have fixed the issue. Indeed tower nextflow made a breaking change. I have started maintaining project again @jfy133 @diegomscoelho