A top-level `bin` folder with hidden files (e.g. `.gitkeep`) results in failure on cloud based runs

[abhi18av]

Steps to reproduce:

1. Create a bin directory and add a .gitkeep file to instruct git to track it (example here https://github.com/abhi18av/hello/tree/empty-bin)

2. Run the pipeline on S3 backed work-dir (Tested the expected behavior on Azure - which is working)

3. The pipeline fails to start with the following error

Error executing process > 'sayHello (3)'

Caused by:
  Essential container in task exited

Command executed:

  echo 'Hello world!'

Command exit status:
  1

Command output:
  (empty)

Command error:
  chmod: /root/nextflow-bin/*: No such file or directory

Work dir:
  /fusion/s3/bucket/scratch/2BOnOd77jYxcoP/0e/e37d1f5c16cd6ffba31bf2f6b52d75

[abhi18av]

Further investigation of the generated .command.run files in the case of AWS s3/fusion backed workdir reveals that the main cause is due to the behavior of chmod command and the default file visibility granted by /* pattern.

Steps to reproduce (on vanilla bash)

1. Create an empty dir

$ mkdir nextflow-bin

2. Add a dummy file

$ touch nextflow-bin/.gitkeep

3. Check current permissions

$ ls nextflow-bin/ -lah
total 0
drwxr-xr-x  3 eklavya staff  96 Sep 22 14:11 .
drwxr-xr-x 10 eklavya staff 320 Sep 22 14:11 ..
-rw-r--r--  1 eklavya staff   0 Sep 22 14:11 .gitkeep

4. Try to elevate permissions of all contents of nextflow-bin (leads to the Error)

$   chmod +x nextflow-bin/*

chmod: cannot access 'nextflow-bin/*': No such file or directory

Observation

This behaviour is not observed in case there is a visible file within nextflow-bin

$ touch nextflow-bin/temp.sh

$ chmod +x nextflow-bin/*

$ ls nextflow-bin/ -lah
total 0
drwxr-xr-x  4 eklavya staff 128 Sep 22 14:09 .
drwxr-xr-x 10 eklavya staff 320 Sep 22 13:55 ..
-rwxr-xr-x  1 eklavya staff   0 Sep 22 13:56 .gitkeep
-rwxr-xr-x  1 eklavya staff   0 Sep 22 14:09 temp.sh

Notice above, the permissions for .gitkeep have also been elevated in addition to the expected temp.sh.

Suggested solution

I think that updating the chmod command to rely upon the -R option which covers hidden as well as visible files, as shown below

$ chmod +x -R ./nextflow-bin/

$ ls nextflow-bin/ -lah
total 0
drwxr-xr-x  3 eklavya staff  96 Sep 22 14:16 .
drwxr-xr-x 10 eklavya staff 320 Sep 22 14:15 ..
-rwxr-xr-x  1 eklavya staff   0 Sep 22 14:16 .gitkeep


$ touch nextflow-bin/temp.sh

$ chmod +x -R ./nextflow-bin/

$ ls -lah nextflow-bin/
total 0
drwxr-xr-x  4 eklavya staff 128 Sep 22 14:18 .
drwxr-xr-x 10 eklavya staff 320 Sep 22 14:15 ..
-rwxr-xr-x  1 eklavya staff   0 Sep 22 14:16 .gitkeep
-rwxr-xr-x  1 eklavya staff   0 Sep 22 14:18 temp.sh

[abhi18av]

@pditommaso @bentsherman @jorgeaguileraseqera , please let me know if the suggestion above makes sense and I'll take this forward.

[jorgeaguileraseqera]

also, you can use

shopt -s dotglob

change files

shopt -u dotglob

[abhi18av]

Ah, this is some advanced bash-fu Jorge! 😄

[pditommaso]

and it crashes!

[abhi18av]

Mmm, but here it seems to be working no?

[pditommaso]

I've patched, don't worry

[abhi18av]