macOS: task input files sporadically fail to mount (zero length)

[jdidion]

The attached workflow returns the correct output sometimes and other times does not. In the failure case it is because the input file is not correctly staged in the task's work/ directory - the staged file is empty rather than containing the correct content. When I set --copy-input-files it always works correctly.

The exact command line I use is:

miniwdl run --error-json --copy-input-files -i input.json -p submodule test.wdl

and input.json is:

{"cat_file.in_txt": "data/in.txt", "cat_file.in_int": 1}

failing_test.zip

[mlin]

@jdidion is this on macOS? Adjusting this setting may help if so https://github.com/chanzuckerberg/miniwdl/issues/145#issuecomment-733435644 -- I am still looking into it

[jdidion]

Yes it is macOS. Which setting exactly? TMPDIR?

[mlin]

@jdidion In this direct comment link https://github.com/chanzuckerberg/miniwdl/issues/145#issuecomment-733435644 @tomkinsc reported that disabling this obscure Docker for Mac setting "Use gRPC FUSE for file sharing" improved things for him (though it may not have been exactly the same problem).

The miniwdl unit test suite (which starts up hundreds of containers over various cases) reliably fails on my MacBook due to this problem, even though it's intermittent for any one. I have been meaning to fiddle with that gRPC setting to see if it helps, but haven't gotten to it yet.

[jdidion]

Got it - I'll try this out when I'm back to work next week.

[mlin]

I can confirm that disabling this setting, Docker for Mac > Preferences > Experimental Features > Use gRPC for file sharing makes the miniwdl unit test suite work well on my MacBook, while it's busted with that setting on as default.

(image credit @tomkinsc; cc @lynnlangit)

One can google a variety of issues people have opened with either alternative of that setting, so I suppose we just have to make do with this pitfall for the time being =/

[nh13]

Chiming in that I hit this too, and using the "legacy osxfs files sharing" fixed it for me too.

[adamnovak]

I ran into this issue using Toil with MiniWDL's Docker task runner. Changing the Docker Desktop file sharing implementation from "gRPC FUSE" to "VirtioFS" seems to have solved it.

[aofarrel]

I can confirm that disabling this setting, Docker for Mac > Preferences > Experimental Features > Use gRPC for file sharing makes the miniwdl unit test suite work well on my MacBook, while it's busted with that setting on as default.

Just popping in to say that as of more recent versions of Docker, this setting has moved to General, and it seems gRPC FUSE might be the default. gRPC FUSE seems to still problematic as stated above, although in my case I got errors even though I was using a test workflow with no inputs at all.