Double cgroup entries in /proc/mounts / ability to mount /dsl within the container

[dmjacobsen]

Originally Reported by Miguel Gila

Hi, I'm getting these sort of warnings coming from a job that reads /proc/mounts:

20 Dec 23:23:09|processes.py| !!WARNING!!2212!! Could not open file /dsl/dev/memory
/dev/memory/slurm/uid_23421/job_19837/step_batch/memory.max_usage_in_bytes: [Errno 2] No such file or directory: '/dsl/dev/memory\n/dev/memory/slurm/uid_23421/job_19837/step_batch/memory.max_usage_in_bytes'

The job does this:

    return commands.getoutput("grep \'^cgroup\' /proc/mounts|grep memory| awk \'{print $2}\'")

Which actually returns:

none /dev/mcgroup cgroup rw,relatime,memory,release_agent=/etc/slurm/cgroup/release_memory 0 0
none /dsl/dev/mcgroup cgroup rw,relatime,memory,release_agent=/etc/slurm/cgroup/release_memory 0 0
cgroup /dsl/dev/memory cgroup rw,nosuid,nodev,noexec,relatime,memory,release_agent=/etc/slurm/cgroup/release_memory 0 0
none /dev/mcgroup cgroup rw,relatime,memory,release_agent=/etc/slurm/cgroup/release_memory 0 0
cgroup /dev/memory cgroup rw,nosuid,nodev,noexec,relatime,memory,release_agent=/etc/slurm/cgroup/release_memory 0 0

Is there any easy way to make sure /proc/mounts is clear of any /dsl entries? Otherwise, could it be possible to mount /dsl in the container? I've tried to add it to udiRoot, but gives an error with the bind-mound routine:

srun shifter  --volume=/dsl:mnt ls -lhaF /
ERROR: unclean exit from bind-mount routine. /var/udiMount//dsl may still be mounted.
BIND MOUNT FAILED from /dsl to /var/udiMount//dsl
   --- REASON: : Invalid argument
FAILED to properly setup site modifications
FAILED to mount image into UDI
FAILED to setup image.
srun: error: nid00035: task 0: Exited with exit code 1
srun: Terminating job step 19868.0

Thanks, Miguel

[dmjacobsen]

I wrote on 2015/12/22:

The user-requested bind mounts (i.e., using --volume) can only perform bind mounts within the container, I think (I'll re-read this shortly, as I'm mucking with related code right now). I do, in fact, often bind-mount /dsl into the container, but it is done while the container is chroot'ed within /dsl (i.e., building a shifter container out of /). It's highly weird but used to provide ccm within native slurm. Anyway, try making /dsl available in your containers by configuring it as a siteFs mount in udiRoot.conf and see if that helps. I'll try to think more deeply about this tomorrow.

[dmjacobsen]

Hi Miguel,

/proc/mounts is probably not the proper file to read in a chroot (and/or container) environment. Try reading /proc/self/mounts instead.

-Doug