girder_worker errors when dockerd is run with --selinux-enabled

[predicative]

Running girder_worker with the docker plugin enabled when dockerd was started with the flag --selinux-enabled results in errors relating to file access and chmod when attempting to run a container. Cf. the output below. Starting dockerd without this flag results in a clean run.

INFO:root:Created LRU Cache for 'tilesource' with 1934 maximum size
WARNING:ctk_cli.module:'reference' attribute of 'file' is not part of the spec yet (CTK issue #623)

>> CLI Parameters ...

Namespace(analysis_mag=20.0, analysis_roi=[14504.0, 17107.0, 767.0, 811.0], analysis_tile_size=4096.0, foreground_threshold=60.0, inputImageFile='/mnt/girder_worker/data/TCGA-02-0010-01Z-00-DX4.07de2e55-a8fe-40ee-9e98-bcb78050b9f7.svs/TCGA-02-0010-01Z-00-DX4.07de2e55-a8fe-40ee-9e98-bcb78050b9f7.svs', local_max_search_radius=10.0, max_radius=30.0, min_fgnd_frac=0.5, min_nucleus_area=80.0, min_radius=20.0, outputNucleiAnnotationFile='/mnt/girder_worker/data/output.anot', reference_mu_lab=[8.63234435, -0.11501964, 0.03868433], reference_std_lab=[0.57506023, 0.10403329, 0.01364062], scheduler_address='', stain_1='hematoxylin', stain_2='eosin', stain_3='null')
Traceback (most recent call last):
  File "NucleiDetection/NucleiDetection.py", line 368, in <module>
    main(CLIArgumentParser().parse_args())
  File "NucleiDetection/NucleiDetection.py", line 182, in main
    raise IOError('Input image file does not exist.')
IOError: Input image file does not exist.
[2017-04-25 10:44:36,516] ERROR: Error setting perms on docker tempdir /home/neal/work/DSA-dev/tmp/tmpYxc3a3.
STDOUT: 
STDERR:chmod: /mnt/girder_worker/data: Permission denied
chmod: /mnt/girder_worker/data: Permission denied

Exception: Docker tempdir chmod returned code 1.
  File "/home/neal/work/DSA-dev/virtualenv/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/neal/work/DSA-dev/virtualenv/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/tasks.py", line 17, in run
    return core.run(*pargs, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/core/utils.py", line 122, in wrapped
    return fn(*args, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/core/__init__.py", line 366, in run
    events.trigger('run.finally', info)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/core/events.py", line 73, in trigger
    handler['handler'](e)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/plugins/docker/__init__.py", line 99, in task_cleanup
    raise Exception('Docker tempdir chmod returned code %d.' % p.returncode)

[danlamanna]

I don't fully remember how SELinux works (or doesn't), but if we were planning on targeting RedHat platforms in the future it might make sense to distribute a policy module along with it. @mathstuf might know more.

[mathstuf]

(Shot in the dark) The --volume option being passed to docker is probably missing a :Z option:

docker run -v /var/db:/var/db:Z rhel7 /bin/sh

[zachmullen]

Thanks @mathstuf , we indeed were not adding that option. Worth trying out to see if using that makes it work with SELinux enabled.

[predicative]

If I change the relevant part of plugins/docker/__init__.py to add the :Z option, the chmod error disappears, but the input file still doesn't exist and now the output file cannot be found (perhaps in relation to the script not finishing because the input isn't found). It still runs clean when leaving out the --selinux-enabled flag with :Z option added.

@@ -87,7 +87,7 @@ def task_cleanup(e):
     if e.info['task']['mode'] == 'docker' and '_tempdir' in e.info['kwargs']:
         tmpdir = e.info['kwargs']['_tempdir']
         cmd = [
-            'docker', 'run', '--rm', '-v', '%s:%s' % (tmpdir, DATA_VOLUME),
+            'docker', 'run', '--rm', '-v', '%s:%s:Z' % (tmpdir, DATA_VOLUME),
             'busybox', 'chmod', '-R', 'a+rw', DATA_VOLUME
         ]
         p = subprocess.Popen(args=cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

INFO:root:Created LRU Cache for 'tilesource' with 1934 maximum size
WARNING:ctk_cli.module:'reference' attribute of 'file' is not part of the spec yet (CTK issue #623)

>> CLI Parameters ...

Namespace(analysis_mag=20.0, analysis_roi=[14175.0, 16923.0, 762.0, 771.0], analysis_tile_size=4096.0, foreground_threshold=60.0, inputImageFile='/mnt/girder_worker/data/TCGA-02-0010-01Z-00-DX4.07de2e55-a8fe-40ee-9e98-bcb78050b9f7.svs/TCGA-02-0010-01Z-00-DX4.07de2e55-a8fe-40ee-9e98-bcb78050b9f7.svs', local_max_search_radius=10.0, max_radius=30.0, min_fgnd_frac=0.5, min_nucleus_area=80.0, min_radius=20.0, outputNucleiAnnotationFile='/mnt/girder_worker/data/output2.anot', reference_mu_lab=[8.63234435, -0.11501964, 0.03868433], reference_std_lab=[0.57506023, 0.10403329, 0.01364062], scheduler_address='', stain_1='hematoxylin', stain_2='eosin', stain_3='null')
Traceback (most recent call last):
  File "NucleiDetection/NucleiDetection.py", line 368, in <module>
    main(CLIArgumentParser().parse_args())
  File "NucleiDetection/NucleiDetection.py", line 182, in main
    raise IOError('Input image file does not exist.')
IOError: Input image file does not exist.
Exception: Output filepath /home/neal/work/DSA-dev/tmp/tmpRJtayd/output2.anot does not exist.
  File "/home/neal/work/DSA-dev/virtualenv/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/neal/work/DSA-dev/virtualenv/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/tasks.py", line 17, in run
    return core.run(*pargs, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/core/utils.py", line 122, in wrapped
    return fn(*args, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/core/__init__.py", line 319, in run
    auto_convert=auto_convert, validate=validate, **kwargs)
  File "/home/neal/work/DSA-dev/girder_worker/girder_worker/plugins/docker/executor.py", line 207, in run
    raise Exception('Output filepath %s does not exist.' % path)