Docker in Windows: `run:` and `bash:` not working

[gaow]

I do not have a Windows machine, but I wonder if you've tried docker run and bash actions on Windows. Earlier I was asking someone to run LaTeX from my docker images on their Windows machine. But it did not work. With run:

[1]
run: container='gaow/debian-texlive'
    pandoc --help

I believe the error message was "exec permission denied", because the docker command looks like docker run ... gaow/debian-texlive /var/lib/sos/xxxxx.bat. That is, it did not call bash but instead tries to run the bat file by itself.

Then I asked him to try with bash:

[1]
bash: container='gaow/debian-texlive'
    pandoc --help

docker correctly used /bin/bash to run the command this time. But the error message was something like xxx.sh: Is directory. I suspect it is because the mount volume command argument, -v .sos/script.sh:/var/lib/sos/script.sh somehow makes docker think it is a directory?

My questions:

1. Can you reproduce those on Windows?
2. Why do we seem to map the script, not its directory, in -v?

[BoPeng]

I suspect that this is what happened.

[gaow]

Interesting ... but this is not the case for him. He uses cygwin on F drive. The path mapping was like -v /F/cygwin...:/F/cygwin..., which looks fine.

[BoPeng]

Is he using docker native or docker toolbox? The problem was with the docker toolbox. From outside it looks ok because /F/cygwin is what you see, but the docker VM only sees what VirtualBox shared to it, so /F/cygwin can be something else.

[gaow]

I see. It might be the problem. I'll ask him to check. I just want to make sure the examples I post do work for Windows if properly configured -- i take that is the case right?

I was reading the documentation RE docker toolbox. I think it might help to point to people recommendation how to install docker on their computer? On Linux for example here is my recommendation:

1. Run commands below:

curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

2. Log out and log back in (no need to reboot computer)

[BoPeng]

I had installation instructions on docker toolbox for mac but everyone is moving to docker native on mac. The same goes with docker for windows although docker native is only available for windows 10+. https://vatlab.github.io/sos-docs/doc/tutorials/SoS_Docker_Guide.html#Common-problems-14 contains some warnings about this toolbox trouble but I guess no one will really read it.

[BoPeng]

Yes, your error message is an indication of mapping an non-existing file into docker, so docker assumes that it is a directory. The non-existing part is what we saw for docker toolbox when the /c/Users/.../script does not exist from docker's eye since C:\Users is mapped by virtual box to /Users so docker can only see the mapped file with path /Users/.../script.

[gaow]

Good to know! But still, the run action for docker in Windows is still problematic right? because the consequence is that it tries to run a .bat command in Linux, as the docker run command shows.

[BoPeng]

run is bat under windows because that is the default script that windows executes. It is arguable if we should change the meaning of run with container because docker windows can execute windows containers, so bat could be the one that is needed anyway.

There is an open ticket for this #1006

[vibbits]

Hi just an addition on our Windows system and Anaconda setup.

bash: docker_image='gaow/debian-texlive'
ls

gives

/var/lib/sos/docker_run_932.sh: line 1: $'ls\r': command not found
Failed to locate specified command (exitcode=127). The script has been saved to C:\Data\software\OmicsIntegrationToolkit\.sos/docker_run_932.sh. To reproduce the error please run:
``docker run --rm   -v /C/Data/software/OmicsIntegrationToolkit:/C/Data/software/OmicsIntegrationToolkit -v /C/Data/software/OmicsIntegrationToolkit/tmpjl3dq39h/docker_run_932.sh:/var/lib/sos/docker_run_932.sh    -t  -w=/C/Data/software/OmicsIntegrationToolkit     gaow/debian-texlive /bin/bash /var/lib/sos/docker_run_932.sh``

I suspected correctly that the line ending in the sh file was Windows style and not Unix style. As soon as I changed this I could execute the command on the command line.

Note: there is an error in the command shown - it says in the red box /C/Data/software/OmicsIntegrationToolkit/tmpjl3dq39h/docker_run_932.sh

but in fact the shell scripts resides in /C/Data/software/OmicsIntegrationToolkit/.sos/docker_run_932.sh as stated in the error message.

[BoPeng]

I see. So without considering #1006 where the container could be windows-based, we should convert all scripts to unix line ending before sending to docker. This makes sense and I will do it in a moment.

For two, sos uses a temporary script for docker execution (and remove it later) so the error message will show a temp directory. We only copy the script to a more accessible location for easier debugging when error happens.

[BoPeng]

@vibbits I pushed a patch but I cannot test it because my docker failed to switch to linux containers.

[vibbits]

I just wanted to test the new version but found that jupter notebook is not able to start the kernel. getting this issue now and kernel is unsuccessfully restarted 'infinitively'

 File "C:\Users\Albot_2\Anaconda3\envs\sos notebook\lib\site-packages\sos_notebook\kernel.py", line 33, in <module>
    from sos.utils import (PrettyRelativeTime, WorkflowDict, env, log_to_file,
ImportError: cannot import name 'PrettyRelativeTime'

[BoPeng]

This is usually a sign of outdated sos, could you run

pip install sos -U

and try again? I have tried to specify required sos version in sos-notebook/setup.py but updating sos-notebook does not seem to automatically update sos. Not sure what else I can do.

[vibbits]

Yes, sos was on the latest version - it stated 0.16.10. I probably should update sos-notebook, too.

[BoPeng]

Yes, you are right. It is sos-notebook's fault to call a utility function that no longer exists in sos. I think this also explains the version dependency problem because your version of sos-notebook depends on some version of sos, and up.

[vibbits]

works fine now after update of sos-notebook - now on 0.16.10 too.

[BoPeng]

Great. We have changed versioning scheme to 3 fields (#1027) and will keep evolving until we reach 1.0.0, after which the API will be much more stable and there will be less compatibility problems like this.

BTW, I still could not test

bash: docker_image='gaow/debian-texlive'
ls

so I would appreciate it if you could confirm if the line-ending problem has been fixed.