Singularity %files behavior change in 3.6 and later

[samcmill]

The behavior of the Singularity %files directive changed beginning in version 3.6. See https://github.com/hpcng/singularity/issues/5514.

For this Singularity definition file:

Bootstrap: docker
From: ubuntu:18.04

%files
   foo /var/tmp

%post
   ls -l /var/tmp/foo

Singularity 3.5.3:

# singularity --version
singularity version 3.5.3

# singularity build foo.sif Singularity.def 
INFO:    Starting build...
INFO:    Copying foo to /tmp/rootfs-f4ee8c55-7790-11eb-b63f-0242ac110002/var/tmp
INFO:    Running post scriptlet
+ ls -l /var/tmp/foo
-rw-r--r--. 1 root root 0 Feb 25 17:43 /var/tmp/foo
INFO:    Creating SIF file...
INFO:    Build complete: foo.sif

Singularity 3.6.4:

# singularity --version
singularity version 3.6.4

# singularity build foo.sif Singularity.def 
INFO:    Starting build...
INFO:    Copying foo to /tmp/build-temp-392680558/rootfs/var/tmp
INFO:    Running post scriptlet
+ ls -l /var/tmp/foo
ls: cannot access '/var/tmp/foo': No such file or directory

Changing the path from /var/tmp to /opt works with either version:

INFO:    Copying foo to /tmp/build-temp-697021000/rootfs/opt
INFO:    Running post scriptlet
+ ls -l /opt/foo
-rw-r--r--. 1 root root 0 Feb 25 17:41 /opt/foo
INFO:    Creating SIF file...
INFO:    Build complete: foo.sif

The --working-directory=/other/path option can be used to generate a Singularity definition file that works around this behavior change.

A better solution might be to use %setup to stage files into the container build environment. Alternatively, use a default location other than /var/tmp or /tmp when copying files from the host (such as the package option in many building blocks).

Originally reported in #341 by @VI-gha

[samcmill]

This is also likely the root cause of #328.