cwltool
cwltool copied to clipboard
Published
20 hours ago •
common-workflow-language
common-workflow-language
improve test running on mac-os
Codecov Report
Merging #1710 (92c4073) into main (92c4073) will not change coverage. The diff coverage is
n/a.
:exclamation: Current head 92c4073 differs from pull request most recent head bbda773. Consider uploading reports for the commit bbda773 to get more accurate results
@@ Coverage Diff @@
## main #1710 +/- ##
=======================================
Coverage 66.60% 66.60%
=======================================
Files 89 89
Lines 15850 15850
Branches 4188 4188
=======================================
Hits 10557 10557
Misses 4203 4203
Partials 1090 1090
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
This pull request fixes 6 alerts when merging e011b4247f214038cec6ee43e0eae3f002353de1 into 8763bed244f4d6a05965e90667020137401b36bb - view on LGTM.com
fixed alerts:
- 6 for Unused import
![lgtm-com[bot] avatar](https://avatars.githubusercontent.com/in/17324?v=4 "Published by a pub.dev verified publisher"){kind=small}
I'm still getting a few test failures:
========================================================================================================================== FAILURES ==========================================================================================================================
________________________________________________________________________________________________________________ TestMpiRun.test_mpi_workflow ________________________________________________________________________________________________________________
[gw4] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10
self = <tests.test_mpi.TestMpiRun object at 0x10cc7d5a0>, fake_mpi_conf = '/Users/jakefennick/cwltool/tmp/popen-gw4/fake_mpi0/plat_mpi.yml', tmp_path = PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw4/test_mpi_workflow0')
@needs_docker
def test_mpi_workflow(self, fake_mpi_conf: str, tmp_path: Path) -> None:
np = 3
input_file = make_processes_input(np, tmp_path)
stdout = StringIO()
stderr = StringIO()
with working_directory(tmp_path):
rc = main(
argsl=cwltool_args(fake_mpi_conf)
+ [get_data("tests/wf/mpi_simple_wf.cwl"), str(input_file)],
stdout=stdout,
stderr=stderr,
)
> assert rc == 0
E assert 1 == 0
tests/test_mpi.py:189: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220815105930
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl'
DEBUG cwltool:workflow_job.py:503 [workflow _2] initialized from file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl
INFO cwltool:workflow_job.py:791 [workflow _2] start
DEBUG cwltool:workflow_job.py:803 [workflow _2] inputs {
"processes": 3
}
DEBUG cwltool:workflow_job.py:625 [workflow _2] job step file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count not ready
INFO cwltool:workflow_job.py:628 [workflow _2] starting step par_pids
DEBUG cwltool:workflow_job.py:756 [step par_pids] job input {
"file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/processes": 3
}
DEBUG cwltool:workflow_job.py:763 [step par_pids] evaluated job input to {
"file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/processes": 3
}
INFO cwltool:workflow_job.py:74 [step par_pids] start
DEBUG cwltool:command_line_tool.py:1022 [job par_pids] initializing from file:///Users/jakefennick/cwltool/tests/wf/mpi_expr.cwl as part of step par_pids
DEBUG cwltool:command_line_tool.py:1030 [job par_pids] {
"processes": 3
}
DEBUG cwltool:command_line_tool.py:1046 [job par_pids] path mappings is {}
DEBUG cwltool:command_line_tool.py:1101 [job par_pids] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "python"
},
{
"position": [
0,
0
],
"datum": "-c"
},
{
"position": [
0,
1
],
"datum": "import os; print(os.getpid())"
}
]
DEBUG cwltool:job.py:222 [job par_pids] initial work dir {}
INFO cwltool:job.py:276 [job par_pids] /private/tmp/docker_tmpc0qvg7s_$ /Users/jakefennick/cwltool/tmp/popen-gw4/fake_mpi0/fake_mpirun \
--num \
3 \
--no-fail \
python \
-c \
'import os; print(os.getpid())' > /private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62
DEBUG cwltool:job.py:561 Could not collect memory usage, job ended before monitoring began.
INFO cwltool:job.py:433 [job par_pids] completed success
DEBUG cwltool:job.py:436 [job par_pids] outputs {
"pids": {
"location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameext": "",
"class": "File",
"checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
"size": 18,
"http://commonwl.org/cwltool#generation": 0
}
}
DEBUG cwltool:workflow_job.py:577 [step par_pids] produced output {
"file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/pids": {
"location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameext": "",
"class": "File",
"checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
"size": 18,
"http://commonwl.org/cwltool#generation": 0
}
}
INFO cwltool:workflow_job.py:587 [step par_pids] completed success
DEBUG cwltool:job.py:464 [job par_pids] Removing input staging directory /private/tmp/docker_tmpvqgs9fql
DEBUG cwltool:job.py:472 [job par_pids] Removing temporary directory /private/tmp/docker_tmpe7y7z9ia
INFO cwltool:workflow_job.py:628 [workflow _2] starting step count
DEBUG cwltool:workflow_job.py:756 [step count] job input {
"file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count/pid_file": {
"location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameext": "",
"class": "File",
"checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
"size": 18,
"http://commonwl.org/cwltool#generation": 0
}
}
DEBUG cwltool:workflow_job.py:763 [step count] evaluated job input to {
"file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count/pid_file": {
"location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameext": "",
"class": "File",
"checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
"size": 18,
"http://commonwl.org/cwltool#generation": 0
}
}
INFO cwltool:workflow_job.py:74 [step count] start
WARNING cwltool:command_line_tool.py:431 MPIRequirement with containers is a beta feature
WARNING cwltool:command_line_tool.py:443 MPI has been required while Docker is hinted, discarding Docker hint(s)
DEBUG cwltool:command_line_tool.py:1022 [job count] initializing from file:///Users/jakefennick/cwltool/tests/wf/mpi_line_count.cwl as part of step count
DEBUG cwltool:command_line_tool.py:1030 [job count] {
"pid_file": {
"location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
"nameext": "",
"class": "File",
"checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
"size": 18,
"http://commonwl.org/cwltool#generation": 0
}
}
DEBUG cwltool:command_line_tool.py:1046 [job count] path mappings is {
"file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62": [
"/private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"/var/lib/cwl/stg814c5222-3ba4-4f09-a3d8-54ec212d3faa/f02fc15c1a50421f8123149541a59ca53ef7ef62",
"File",
true
]
}
DEBUG cwltool:command_line_tool.py:1101 [job count] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "wc"
},
{
"position": [
0,
0
],
"datum": "-l"
}
]
ERROR cwltool:executors.py:261 Got workflow error
Traceback (most recent call last):
File "/Users/jakefennick/cwltool/cwltool/executors.py", line 251, in run_jobs
job.run(runtime_context)
File "/Users/jakefennick/cwltool/cwltool/job.py", line 581, in run
self._setup(runtimeContext)
File "/Users/jakefennick/cwltool/cwltool/job.py", line 191, in _setup
os.makedirs(self.outdir)
File "/Users/jakefennick/opt/anaconda3/envs/cwltool_test/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/ATmsgV'
ERROR cwltool:main.py:1409 Workflow error, try again with --debug for more information:
[Errno 30] Read-only file system: '/ATmsgV'
________________________________________________________________________________________________________ test_single_process_subwf_subwf_inline_step _________________________________________________________________________________________________________
[gw5] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10
@needs_docker
def test_single_process_subwf_subwf_inline_step() -> None:
"""Test --single-process on an inline sub-sub-workflow step."""
err_code, stdout, stderr = get_main_output(
[
"--single-process",
"step1/stepX/stepY",
get_data("tests/subgraph/count-lines17-wf.cwl.json"),
get_data("tests/wf/wc-job.json"),
]
)
> assert err_code == 0
E assert 1 == 0
tests/test_subgraph.py:254: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220815105930
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json' to 'file:///Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json'
ERROR cwltool:main.py:1227 Tool definition failed validation:
while scanning for the next token
tests/subgraph/count-lines17-wf.cwl.json:116:1: found character '\t' that cannot start any token
___________________________________________________________________________________________________________ test_print_targets_embedded_sub_subwfs ___________________________________________________________________________________________________________
[gw2] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10
@needs_docker
def test_print_targets_embedded_sub_subwfs() -> None:
"""Confirm that --print-targets works with inline sub-sub-workflows."""
err_code, stdout, stderr = get_main_output(
[
"--print-targets",
get_data("tests/subgraph/count-lines17-wf.cwl.json"),
]
)
> assert err_code == 0
E assert 1 == 0
tests/test_subgraph.py:363: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220815105930
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json' to 'file:///Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json'
ERROR cwltool:main.py:1227 Tool definition failed validation:
while scanning for the next token
tests/subgraph/count-lines17-wf.cwl.json:116:1: found character '\t' that cannot start any token
FYI the two errors about tabs can be fixed by replacing the tabs with spaces in tests/subgraph/count-lines17-wf.cwl.json.
The other error about Docker and MPI being mutually exclusive seems to be hardcoded in the decision tree starting at line 430 of command_line_tool.py I'm not exactly sure what you want to do to fix that. I tried swapping the hints and requirements in tests/wf/mpi_line_count.cwl which (as per the decision tree) ignores MPI. That 'fixes' that test, but note that you will also need to make the same writable=volume.staged change I made in #1708 to get test_iwdr_permutations to pass. WIth those changes, it works on my machine.
diff --git a/cwltool/docker.py b/cwltool/docker.py
index 1d7c9f97..98fe13ad 100644
--- a/cwltool/docker.py
+++ b/cwltool/docker.py
@@ -260,7 +260,9 @@ class DockerCommandLineJob(ContainerCommandLineJob):
"""Append volume a file/dir mapping to the runtime option list."""
if not volume.resolved.startswith("_:"):
_check_docker_machine_path(volume.resolved)
- self.append_volume(runtime, volume.resolved, volume.target)
+ self.append_volume(
+ runtime, volume.resolved, volume.target, writable=volume.staged
+ )
def add_writable_file_volume(
self,
diff --git a/tests/subgraph/count-lines17-wf.cwl.json b/tests/subgraph/count-lines17-wf.cwl.json
index fa0905ef..279790b8 100644
--- a/tests/subgraph/count-lines17-wf.cwl.json
+++ b/tests/subgraph/count-lines17-wf.cwl.json
@@ -113,10 +113,10 @@
"-l"
],
"class": "CommandLineTool",
- "hints": {
- "DockerRequirement": {
- "dockerPull": "docker.io/debian:stable-slim"
- }
+ "hints": {
+ "DockerRequirement": {
+ "dockerPull": "docker.io/debian:stable-slim"
+ }
},
"id": "count-lines17-wf.cwl@step_step1@run@step_stepX@run@step_stepY@run",
"inputs": [
diff --git a/tests/wf/mpi_line_count.cwl b/tests/wf/mpi_line_count.cwl
index a211506f..eb570280 100644
--- a/tests/wf/mpi_line_count.cwl
+++ b/tests/wf/mpi_line_count.cwl
@@ -3,7 +3,7 @@
cwlVersion: v1.1
class: CommandLineTool
-hints:
+requirements:
DockerRequirement:
dockerPull: docker.io/debian:stable-slim
@@ -15,7 +15,7 @@ doc: |
baseCommand: wc
-requirements:
+hints:
cwltool:MPIRequirement:
processes: 1
This pull request fixes 6 alerts when merging c047adbcb1e652bba0d291bb98dc93e8bd560ef4 into f3a35f394697f23281c16cf257d18683d7646178 - view on LGTM.com
fixed alerts:
- 6 for Unused import
This pull request fixes 6 alerts when merging 64ed57323a814b988cd720383800620244acc4a8 into f3a35f394697f23281c16cf257d18683d7646178 - view on LGTM.com
fixed alerts:
- 6 for Unused import
This pull request fixes 6 alerts when merging 2df445210728eca12f9e2e2b51effbb3dea6f2e6 into f3a35f394697f23281c16cf257d18683d7646178 - view on LGTM.com
fixed alerts:
- 6 for Unused import
Thanks @jfennick for the testing; alas docker is not included on the GitHub hosted action runners due to licensing issues and the popular community provided action for install docker on macos is not reliably working right now. Can you test this branch one more time with docker on your macos system?
This pull request fixes 6 alerts when merging 6e10ec6d638fece102721eebe5b0f4872c2fe0f0 into f3a35f394697f23281c16cf257d18683d7646178 - view on LGTM.com
fixed alerts:
- 6 for Unused import
pytest now works on my machine, but tox still gives 3 failures:
========================================================================================================================== FAILURES ==========================================================================================================================
___________________________________________________________________________________________________________________ test_iwdr_permutations ___________________________________________________________________________________________________________________
[gw3] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python
tmp_path_factory = TempPathFactory(_given_basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'), _trace=<pluggy._tracing.TagTracerSub object at 0x1047e80d0>, _basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'))
@needs_docker
def test_iwdr_permutations(tmp_path_factory: Any) -> None:
misc = tmp_path_factory.mktemp("misc")
fifth = misc / "fifth"
fifth.mkdir()
sixth = misc / "sixth"
sixth.mkdir()
seventh = misc / "seventh"
seventh.mkdir()
eighth = misc / "eighth"
eighth.mkdir()
first = misc / "first"
first.touch()
second = misc / "second"
second.touch()
third = misc / "third"
third.touch()
fourth = misc / "fourth"
fourth.touch()
eleventh = misc / "eleventh"
eleventh.touch()
twelfth = misc / "twelfth"
twelfth.touch()
outdir = str(tmp_path_factory.mktemp("outdir"))
err_code, stdout, _ = get_main_output(
[
"--outdir",
outdir,
"--debug",
get_data("tests/wf/iwdr_permutations.cwl"),
"--first",
str(first),
"--second",
str(second),
"--third",
str(third),
"--fourth",
str(fourth),
"--fifth",
str(fifth),
"--sixth",
str(sixth),
"--seventh",
str(seventh),
"--eighth",
str(eighth),
"--eleventh",
str(eleventh),
"--eleventh",
str(twelfth),
]
)
assert err_code == 0
log = json.loads(stdout)["log"]
> assert log["checksum"] == "sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f", log
E AssertionError: {'basename': 'log.txt', 'checksum': 'sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131', 'class': 'File', 'location': 'file:///Users/jakefennick/cwltool/tmp/popen-gw3/outdir0/docker_tmpiw1z7hc3/log.txt', ...}
E assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131
tests/test_iwdr.py:134: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/debian
Digest: sha256:82bab30ed448b8e2509aabe21f40f0607d905b7fd0dec72802627a20274eba55
Status: Image is up to date for debian:latest
docker.io/library/debian:latest
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220817095406
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl'
DEBUG cwltool:main.py:472 Parsed job order from command line: {
"id": "/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl",
"eighth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth"
},
"eleventh": [
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh"
},
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth"
}
],
"fifth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth"
},
"first": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first"
},
"fourth": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth"
},
"ninth": {
"class": "Directory",
"basename": "foo",
"listing": []
},
"second": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second"
},
"seventh": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh"
},
"sixth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth"
},
"tenth": {
"class": "Directory",
"basename": "bar",
"listing": []
},
"third": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third"
}
}
DEBUG cwltool:command_line_tool.py:1022 [job iwdr_permutations.cwl] initializing from file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl
DEBUG cwltool:command_line_tool.py:1030 [job iwdr_permutations.cwl] {
"eighth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
"basename": "eighth"
},
"eleventh": [
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
"size": 0,
"basename": "eleventh",
"nameroot": "eleventh",
"nameext": ""
},
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
"size": 0,
"basename": "twelfth",
"nameroot": "twelfth",
"nameext": ""
}
],
"fifth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
"basename": "fifth"
},
"first": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
"size": 0,
"basename": "first",
"nameroot": "first",
"nameext": ""
},
"fourth": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
"size": 0,
"basename": "fourth",
"nameroot": "fourth",
"nameext": ""
},
"ninth": {
"class": "Directory",
"basename": "foo",
"listing": [],
"location": "_:09f0ac62-f068-4c6c-b081-abec23241f5f"
},
"second": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
"size": 0,
"basename": "second",
"nameroot": "second",
"nameext": ""
},
"seventh": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
"basename": "seventh"
},
"sixth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
"basename": "sixth"
},
"tenth": {
"class": "Directory",
"basename": "bar",
"listing": [],
"location": "_:f6443482-d428-4988-9b8e-32aa9d4a546a"
},
"third": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
"size": 0,
"basename": "third",
"nameroot": "third",
"nameext": ""
}
}
DEBUG cwltool:command_line_tool.py:1046 [job iwdr_permutations.cwl] path mappings is {
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
"/my_other_path/eighth_read_only_directory",
"Directory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
"/YWPMVO/eleventh",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
"/YWPMVO/twelfth",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
"/YWPMVO/fifth_writable_directory",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
"/YWPMVO/first_writable_file",
"WritableFile",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
"/my_other_path/fourth_read_only_file",
"File",
false
],
"_:09f0ac62-f068-4c6c-b081-abec23241f5f": [
"_:09f0ac62-f068-4c6c-b081-abec23241f5f",
"/YWPMVO/nineth_writable_directory_literal",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
"/YWPMVO/second_read_only_file",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
"/my_path/seventh_writable_directory",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
"/YWPMVO/sixth_read_only_directory",
"Directory",
false
],
"_:f6443482-d428-4988-9b8e-32aa9d4a546a": [
"_:f6443482-d428-4988-9b8e-32aa9d4a546a",
"/my_path/tenth_writable_directory_literal",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
"/my_path/third_writable_file",
"WritableFile",
false
]
}
DEBUG cwltool:command_line_tool.py:1101 [job iwdr_permutations.cwl] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "bash"
},
{
"position": [
-1000000,
1
],
"datum": "-c"
},
{
"position": [
0,
0
],
"datum": "find . | sort\nfind /my_path | sort\nfind /my_other_path | sort\necho \"a\" > first_writable_file\necho \"b\" > /my_path/third_writable_file\ntouch fifth_writable_directory/c\ntouch /my_path/seventh_writable_directory/d\nfind . | sort\nfind /my_path | sort\nfind /my_other_path | sort\n"
}
]
INFO cwltool:docker.py:155 ['docker', 'pull', 'docker.io/debian']
DEBUG cwltool:job.py:222 [job iwdr_permutations.cwl] initial work dir {
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
"/YWPMVO/first_writable_file",
"WritableFile",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
"/YWPMVO/second_read_only_file",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
"/my_path/third_writable_file",
"WritableFile",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
"/my_other_path/fourth_read_only_file",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
"/YWPMVO/fifth_writable_directory",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
"/YWPMVO/sixth_read_only_directory",
"Directory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
"/my_path/seventh_writable_directory",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
"/my_other_path/eighth_read_only_directory",
"Directory",
true
],
"_:09f0ac62-f068-4c6c-b081-abec23241f5f": [
"_:09f0ac62-f068-4c6c-b081-abec23241f5f",
"/YWPMVO/nineth_writable_directory_literal",
"WritableDirectory",
true
],
"_:f6443482-d428-4988-9b8e-32aa9d4a546a": [
"_:f6443482-d428-4988-9b8e-32aa9d4a546a",
"/my_path/tenth_writable_directory_literal",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
"/YWPMVO/eleventh",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
"/YWPMVO/twelfth",
"File",
true
],
"_:165769e2-d381-49f7-ba3c-3a5321bd144b": [
"baz",
"/my_path/my_file_literal",
"CreateFile",
true
]
}
INFO cwltool:job.py:276 [job iwdr_permutations.cwl] /private/tmp/docker_tmpiw1z7hc3$ docker \
run \
-i \
--mount=type=bind,source=/private/tmp/docker_tmpiw1z7hc3,target=/YWPMVO \
--mount=type=bind,source=/private/tmp/docker_tmpezipi7as,target=/tmp \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second,target=/YWPMVO/second_read_only_file,readonly \
--mount=type=bind,source=/private/tmp/docker_tmpp_cwer51/third,target=/my_path/third_writable_file \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth,target=/my_other_path/fourth_read_only_file,readonly \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth,target=/YWPMVO/sixth_read_only_directory,readonly \
--mount=type=bind,source=/private/tmp/docker_tmpdec4fwng/seventh,target=/my_path/seventh_writable_directory \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth,target=/my_other_path/eighth_read_only_directory,readonly \
--mount=type=bind,source=/private/tmp/docker_tmprov2ay4z/tenth_writable_directory_literal,target=/my_path/tenth_writable_directory_literal \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh,target=/YWPMVO/eleventh,readonly \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth,target=/YWPMVO/twelfth,readonly \
--mount=type=bind,source=/private/tmp/docker_tmp01btz33i/my_file_literal,target=/my_path/my_file_literal,readonly \
--workdir=/YWPMVO \
--read-only=true \
--net=none \
--log-driver=none \
--user=502:20 \
--rm \
--cidfile=/private/tmp/docker_tmpqxubkvvy/20220817090758-196751.cid \
--env=TMPDIR=/tmp \
--env=HOME=/YWPMVO \
--env=LC_ALL=C \
docker.io/debian \
bash \
-c \
'find . | sort
find /my_path | sort
find /my_other_path | sort
echo "a" > first_writable_file
echo "b" > /my_path/third_writable_file
touch fifth_writable_directory/c
touch /my_path/seventh_writable_directory/d
find . | sort
find /my_path | sort
find /my_other_path | sort
' > /private/tmp/docker_tmpiw1z7hc3/log.txt
INFO cwltool:job.py:943 [job iwdr_permutations.cwl] Max memory used: 0MiB
INFO cwltool:job.py:433 [job iwdr_permutations.cwl] completed success
DEBUG cwltool:job.py:436 [job iwdr_permutations.cwl] outputs {
"log": {
"location": "file:///private/tmp/docker_tmpiw1z7hc3/log.txt",
"basename": "log.txt",
"nameroot": "log",
"nameext": ".txt",
"class": "File",
"checksum": "sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131",
"size": 983,
"http://commonwl.org/cwltool#generation": 0
},
"out": {
"location": "file:///private/tmp/docker_tmpiw1z7hc3",
"basename": "docker_tmpiw1z7hc3",
"nameroot": "docker_tmpiw1z7hc3",
"nameext": "",
"class": "Directory"
}
}
DEBUG cwltool:job.py:464 [job iwdr_permutations.cwl] Removing input staging directory /private/tmp/docker_tmpx38wc7cw
DEBUG cwltool:job.py:472 [job iwdr_permutations.cwl] Removing temporary directory /private/tmp/docker_tmpezipi7as
DEBUG cwltool:process.py:371 Moving /private/tmp/docker_tmpiw1z7hc3 to /Users/jakefennick/cwltool/tmp/popen-gw3/outdir0/docker_tmpiw1z7hc3
INFO cwltool:main.py:1393 Final process status is success
_______________________________________________________________________________________________________________ test_iwdr_permutations_inplace _______________________________________________________________________________________________________________
[gw3] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python
tmp_path_factory = TempPathFactory(_given_basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'), _trace=<pluggy._tracing.TagTracerSub object at 0x1047e80d0>, _basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'))
@needs_docker
def test_iwdr_permutations_inplace(tmp_path_factory: Any) -> None:
misc = tmp_path_factory.mktemp("misc")
fifth = misc / "fifth"
fifth.mkdir()
sixth = misc / "sixth"
sixth.mkdir()
seventh = misc / "seventh"
seventh.mkdir()
eighth = misc / "eighth"
eighth.mkdir()
first = misc / "first"
first.touch()
second = misc / "second"
second.touch()
third = misc / "third"
third.touch()
fourth = misc / "fourth"
fourth.touch()
eleventh = misc / "eleventh"
eleventh.touch()
twelfth = misc / "twelfth"
twelfth.touch()
outdir = str(tmp_path_factory.mktemp("outdir"))
err_code, stdout, _ = get_main_output(
[
"--outdir",
outdir,
"--enable-ext",
"--overrides",
get_data("tests/wf/iwdr_permutations_inplace.yml"),
get_data("tests/wf/iwdr_permutations.cwl"),
"--first",
str(first),
"--second",
str(second),
"--third",
str(third),
"--fourth",
str(fourth),
"--fifth",
str(fifth),
"--sixth",
str(sixth),
"--seventh",
str(seventh),
"--eighth",
str(eighth),
"--eleventh",
str(eleventh),
"--eleventh",
str(twelfth),
]
)
assert err_code == 0
log = json.loads(stdout)["log"]
> assert log["checksum"] == "sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f", log
E AssertionError: {'basename': 'log.txt', 'checksum': 'sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131', 'class': 'File', 'location': 'file:///Users/jakefennick/cwltool/tmp/popen-gw3/outdir2/docker_tmp864pxgrh/log.txt', ...}
E assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131
tests/test_iwdr.py:243: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220817095406
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl'
DEBUG cwltool:main.py:472 Parsed job order from command line: {
"id": "/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl",
"eighth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth"
},
"eleventh": [
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh"
},
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth"
}
],
"fifth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth"
},
"first": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first"
},
"fourth": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth"
},
"ninth": {
"class": "Directory",
"basename": "foo",
"listing": []
},
"second": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second"
},
"seventh": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh"
},
"sixth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth"
},
"tenth": {
"class": "Directory",
"basename": "bar",
"listing": []
},
"third": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third"
}
}
DEBUG cwltool:command_line_tool.py:1022 [job iwdr_permutations.cwl_2] initializing from file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl
DEBUG cwltool:command_line_tool.py:1030 [job iwdr_permutations.cwl_2] {
"eighth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
"basename": "eighth"
},
"eleventh": [
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
"size": 0,
"basename": "eleventh",
"nameroot": "eleventh",
"nameext": ""
},
{
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
"size": 0,
"basename": "twelfth",
"nameroot": "twelfth",
"nameext": ""
}
],
"fifth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
"basename": "fifth"
},
"first": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
"size": 0,
"basename": "first",
"nameroot": "first",
"nameext": ""
},
"fourth": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
"size": 0,
"basename": "fourth",
"nameroot": "fourth",
"nameext": ""
},
"ninth": {
"class": "Directory",
"basename": "foo",
"listing": [],
"location": "_:d7c44520-b263-4ce1-977e-b0a55d92f303"
},
"second": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
"size": 0,
"basename": "second",
"nameroot": "second",
"nameext": ""
},
"seventh": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
"basename": "seventh"
},
"sixth": {
"class": "Directory",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
"basename": "sixth"
},
"tenth": {
"class": "Directory",
"basename": "bar",
"listing": [],
"location": "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6"
},
"third": {
"class": "File",
"location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
"size": 0,
"basename": "third",
"nameroot": "third",
"nameext": ""
}
}
DEBUG cwltool:command_line_tool.py:1046 [job iwdr_permutations.cwl_2] path mappings is {
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
"/my_other_path/eighth_read_only_directory",
"Directory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
"/YWPMVO/eleventh",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
"/YWPMVO/twelfth",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
"/YWPMVO/fifth_writable_directory",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
"/YWPMVO/first_writable_file",
"WritableFile",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
"/my_other_path/fourth_read_only_file",
"File",
false
],
"_:d7c44520-b263-4ce1-977e-b0a55d92f303": [
"_:d7c44520-b263-4ce1-977e-b0a55d92f303",
"/YWPMVO/nineth_writable_directory_literal",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
"/YWPMVO/second_read_only_file",
"File",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
"/my_path/seventh_writable_directory",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
"/YWPMVO/sixth_read_only_directory",
"Directory",
false
],
"_:1aed930b-811c-4a76-aec7-d4e347e3f5a6": [
"_:1aed930b-811c-4a76-aec7-d4e347e3f5a6",
"/my_path/tenth_writable_directory_literal",
"WritableDirectory",
false
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
"/my_path/third_writable_file",
"WritableFile",
false
]
}
DEBUG cwltool:command_line_tool.py:1101 [job iwdr_permutations.cwl_2] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "bash"
},
{
"position": [
-1000000,
1
],
"datum": "-c"
},
{
"position": [
0,
0
],
"datum": "find . | sort\nfind /my_path | sort\nfind /my_other_path | sort\necho \"a\" > first_writable_file\necho \"b\" > /my_path/third_writable_file\ntouch fifth_writable_directory/c\ntouch /my_path/seventh_writable_directory/d\nfind . | sort\nfind /my_path | sort\nfind /my_other_path | sort\n"
}
]
DEBUG cwltool:job.py:222 [job iwdr_permutations.cwl_2] initial work dir {
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
"/YWPMVO/first_writable_file",
"WritableFile",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
"/YWPMVO/second_read_only_file",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
"/my_path/third_writable_file",
"WritableFile",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
"/my_other_path/fourth_read_only_file",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
"/YWPMVO/fifth_writable_directory",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
"/YWPMVO/sixth_read_only_directory",
"Directory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
"/my_path/seventh_writable_directory",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
"/my_other_path/eighth_read_only_directory",
"Directory",
true
],
"_:d7c44520-b263-4ce1-977e-b0a55d92f303": [
"_:d7c44520-b263-4ce1-977e-b0a55d92f303",
"/YWPMVO/nineth_writable_directory_literal",
"WritableDirectory",
true
],
"_:1aed930b-811c-4a76-aec7-d4e347e3f5a6": [
"_:1aed930b-811c-4a76-aec7-d4e347e3f5a6",
"/my_path/tenth_writable_directory_literal",
"WritableDirectory",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
"/YWPMVO/eleventh",
"File",
true
],
"file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth": [
"/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
"/YWPMVO/twelfth",
"File",
true
],
"_:1a3b779e-91c7-41d1-844c-c118a17aba55": [
"baz",
"/my_path/my_file_literal",
"CreateFile",
true
]
}
INFO cwltool:job.py:276 [job iwdr_permutations.cwl_2] /private/tmp/docker_tmp864pxgrh$ docker \
run \
-i \
--mount=type=bind,source=/private/tmp/docker_tmp864pxgrh,target=/YWPMVO \
--mount=type=bind,source=/private/tmp/docker_tmp65c8saw7,target=/tmp \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second,target=/YWPMVO/second_read_only_file,readonly \
--mount=type=bind,source=/private/tmp/docker_tmp4cw3po0e/third,target=/my_path/third_writable_file \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth,target=/my_other_path/fourth_read_only_file,readonly \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth,target=/YWPMVO/sixth_read_only_directory,readonly \
--mount=type=bind,source=/private/tmp/docker_tmphhcsd5f5/seventh,target=/my_path/seventh_writable_directory \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth,target=/my_other_path/eighth_read_only_directory,readonly \
--mount=type=bind,source=/private/tmp/docker_tmpubgq0gqh/tenth_writable_directory_literal,target=/my_path/tenth_writable_directory_literal \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh,target=/YWPMVO/eleventh,readonly \
--mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth,target=/YWPMVO/twelfth,readonly \
--mount=type=bind,source=/private/tmp/docker_tmpqgsqof9n/my_file_literal,target=/my_path/my_file_literal,readonly \
--workdir=/YWPMVO \
--read-only=true \
--net=none \
--log-driver=none \
--user=502:20 \
--rm \
--cidfile=/private/tmp/docker_tmplagziroh/20220817090803-686776.cid \
--env=TMPDIR=/tmp \
--env=HOME=/YWPMVO \
--env=LC_ALL=C \
docker.io/debian \
bash \
-c \
'find . | sort
find /my_path | sort
find /my_other_path | sort
echo "a" > first_writable_file
echo "b" > /my_path/third_writable_file
touch fifth_writable_directory/c
touch /my_path/seventh_writable_directory/d
find . | sort
find /my_path | sort
find /my_other_path | sort
' > /private/tmp/docker_tmp864pxgrh/log.txt
INFO cwltool:job.py:943 [job iwdr_permutations.cwl_2] Max memory used: 0MiB
INFO cwltool:job.py:433 [job iwdr_permutations.cwl_2] completed success
DEBUG cwltool:job.py:436 [job iwdr_permutations.cwl_2] outputs {
"log": {
"location": "file:///private/tmp/docker_tmp864pxgrh/log.txt",
"basename": "log.txt",
"nameroot": "log",
"nameext": ".txt",
"class": "File",
"checksum": "sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131",
"size": 983,
"http://commonwl.org/cwltool#generation": 0
},
"out": {
"location": "file:///private/tmp/docker_tmp864pxgrh",
"basename": "docker_tmp864pxgrh",
"nameroot": "docker_tmp864pxgrh",
"nameext": "",
"class": "Directory"
}
}
DEBUG cwltool:job.py:464 [job iwdr_permutations.cwl_2] Removing input staging directory /private/tmp/docker_tmppdvy1ol_
DEBUG cwltool:job.py:472 [job iwdr_permutations.cwl_2] Removing temporary directory /private/tmp/docker_tmp65c8saw7
DEBUG cwltool:process.py:371 Moving /private/tmp/docker_tmp864pxgrh to /Users/jakefennick/cwltool/tmp/popen-gw3/outdir2/docker_tmp864pxgrh
INFO cwltool:main.py:1393 Final process status is success
____________________________________________________________________________________________________________________ test_cid_file_dir[] _____________________________________________________________________________________________________________________
[gw1] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python
tmp_path = PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0'), factor = ''
@needs_docker
@pytest.mark.parametrize("factor", test_factors)
def test_cid_file_dir(tmp_path: Path, factor: str) -> None:
"""Test --cidfile-dir option works."""
test_file = "cache_test_workflow.cwl"
with working_directory(tmp_path):
commands = factor.split()
commands.extend(
["--cidfile-dir", str(tmp_path), get_data("tests/wf/" + test_file)]
)
error_code, stdout, stderr = get_main_output(commands)
stderr = re.sub(r"\s\s+", " ", stderr)
assert "completed success" in stderr
assert error_code == 0
cidfiles_count = sum(1 for _ in tmp_path.glob("**/*"))
> assert cidfiles_count == 2
E assert 5 == 2
tests/test_examples.py:1124: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Status: Image is up to date for alpine:latest
docker.io/library/alpine:latest
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO cwltool:main.py:1046 3.1.20220817095406
INFO cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl'
DEBUG cwltool:main.py:472 Parsed job order from command line: {
"id": "/Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl"
}
DEBUG cwltool:workflow_job.py:503 [workflow _10] initialized from file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl
INFO cwltool:workflow_job.py:791 [workflow _10] start
DEBUG cwltool:workflow_job.py:803 [workflow _10] inputs {}
INFO cwltool:workflow_job.py:628 [workflow _10] starting step task1
DEBUG cwltool:workflow_job.py:756 [step task1] job input {
"file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task1/message": "one"
}
DEBUG cwltool:workflow_job.py:763 [step task1] evaluated job input to {
"file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task1/message": "one"
}
INFO cwltool:workflow_job.py:74 [step task1] start
DEBUG cwltool:command_line_tool.py:1022 [job task1] initializing from file:///Users/jakefennick/cwltool/tests/wf/touch_tool.cwl as part of step task1
DEBUG cwltool:command_line_tool.py:1030 [job task1] {
"message": "one"
}
DEBUG cwltool:command_line_tool.py:1046 [job task1] path mappings is {}
DEBUG cwltool:command_line_tool.py:1101 [job task1] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "touch"
},
{
"position": [
1,
"message"
],
"datum": "one"
}
]
INFO cwltool:docker.py:155 ['docker', 'pull', 'docker.io/alpine']
DEBUG cwltool:job.py:222 [job task1] initial work dir {}
INFO cwltool:job.py:276 [job task1] /private/tmp/docker_tmp7uak_pw8$ docker \
run \
-i \
--mount=type=bind,source=/private/tmp/docker_tmp7uak_pw8,target=/WXkvbN \
--mount=type=bind,source=/private/tmp/docker_tmpmvbb_ve4,target=/tmp \
--workdir=/WXkvbN \
--read-only=true \
--user=502:20 \
--rm \
--cidfile=/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220817090807-856622.cid \
--env=TMPDIR=/tmp \
--env=HOME=/WXkvbN \
docker.io/alpine \
touch \
one
INFO cwltool:job.py:943 [job task1] Max memory used: 0MiB
INFO cwltool:job.py:433 [job task1] completed success
DEBUG cwltool:job.py:436 [job task1] outputs {
"out": {
"location": "file:///private/tmp/docker_tmp7uak_pw8/one",
"basename": "one",
"nameroot": "one",
"nameext": "",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"http://commonwl.org/cwltool#generation": 0
}
}
DEBUG cwltool:workflow_job.py:577 [step task1] produced output {}
INFO cwltool:workflow_job.py:587 [step task1] completed success
DEBUG cwltool:job.py:464 [job task1] Removing input staging directory /private/tmp/docker_tmpp43ki28h
DEBUG cwltool:job.py:472 [job task1] Removing temporary directory /private/tmp/docker_tmpmvbb_ve4
INFO cwltool:workflow_job.py:628 [workflow _10] starting step task2
DEBUG cwltool:workflow_job.py:756 [step task2] job input {
"file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task2/message": "two"
}
DEBUG cwltool:workflow_job.py:763 [step task2] evaluated job input to {
"file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task2/message": "two"
}
INFO cwltool:workflow_job.py:74 [step task2] start
DEBUG cwltool:command_line_tool.py:1022 [job task2] initializing from file:///Users/jakefennick/cwltool/tests/wf/touch_tool.cwl as part of step task2
DEBUG cwltool:command_line_tool.py:1030 [job task2] {
"message": "two"
}
DEBUG cwltool:command_line_tool.py:1046 [job task2] path mappings is {}
DEBUG cwltool:command_line_tool.py:1101 [job task2] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "touch"
},
{
"position": [
1,
"message"
],
"datum": "two"
}
]
DEBUG cwltool:job.py:222 [job task2] initial work dir {}
INFO cwltool:job.py:276 [job task2] /private/tmp/docker_tmpogdab0mc$ docker \
run \
-i \
--mount=type=bind,source=/private/tmp/docker_tmpogdab0mc,target=/WXkvbN \
--mount=type=bind,source=/private/tmp/docker_tmpiwl2mrok,target=/tmp \
--workdir=/WXkvbN \
--read-only=true \
--user=502:20 \
--rm \
--cidfile=/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220817090808-893250.cid \
--env=TMPDIR=/tmp \
--env=HOME=/WXkvbN \
docker.io/alpine \
touch \
two
INFO cwltool:job.py:943 [job task2] Max memory used: 0MiB
INFO cwltool:job.py:433 [job task2] completed success
DEBUG cwltool:job.py:436 [job task2] outputs {
"out": {
"location": "file:///private/tmp/docker_tmpogdab0mc/two",
"basename": "two",
"nameroot": "two",
"nameext": "",
"class": "File",
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"size": 0,
"http://commonwl.org/cwltool#generation": 0
}
}
DEBUG cwltool:workflow_job.py:577 [step task2] produced output {}
INFO cwltool:workflow_job.py:587 [step task2] completed success
INFO cwltool:workflow_job.py:549 [workflow _10] completed success
DEBUG cwltool:workflow_job.py:551 [workflow _10] outputs {}
DEBUG cwltool:job.py:464 [job task2] Removing input staging directory /private/tmp/docker_tmp_5x9uv03
DEBUG cwltool:job.py:472 [job task2] Removing temporary directory /private/tmp/docker_tmpiwl2mrok
DEBUG cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmp7uak_pw8
DEBUG cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmpw197uwi0
DEBUG cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmpogdab0mc
INFO cwltool:main.py:1393 Final process status is success
This pull request fixes 6 alerts when merging 9a62868103d3c0467046a3d996bbdece601f79be into 79fd3630cf750d178176d5a450dd23938d7fe628 - view on LGTM.com
fixed alerts:
- 6 for Unused import
@jfennick @kinow I've improved test failure reporting for those tests; they should show the unexpected output
This pull request fixes 6 alerts when merging 577cc29d4fbeed6ce08a7bb27eb8cf16db5a4c4d into 79fd3630cf750d178176d5a450dd23938d7fe628 - view on LGTM.com
fixed alerts:
- 6 for Unused import
This pull request fixes 6 alerts when merging 369eddde2e525e9d1a13d34b194112acc21abdc3 into 49fc1c3a1a19eca45d5bce3d1f3ff740ea30ad79 - view on LGTM.com
fixed alerts:
- 6 for Unused import
This pull request fixes 6 alerts when merging c3085377a33a64b67a4f007f4df8825853be8684 into 0e2ced53f8cd2652c4117af697fe2f5c6c94fcb7 - view on LGTM.com
fixed alerts:
- 6 for Unused import
FYI now I'm getting a pytest failure on test_bioconda:
UnavailableInvalidChannel: The channel is not accessible or is invalid.
channel name: pkgs/main
channel url: https://repo.anaconda.com/pkgs/main
error code: 404
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.
Traceback (most recent call last):
File "/private/tmp/docker_tmpeuhhq6t1/run_job.py", line 101, in <module>
sys.exit(main(sys.argv))
File "/private/tmp/docker_tmpeuhhq6t1/run_job.py", line 78, in main
sp = subprocess.Popen( # nosec
File "/Users/jakefennick/opt/anaconda3/envs/lol/lib/python3.10/subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/jakefennick/opt/anaconda3/envs/lol/lib/python3.10/subprocess.py", line 1845, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'seqtk'
The problem is my .condarc file was
channels:
- conda-forge
- omnia
- defaults
The pytest failure goes away if I add
channels:
- conda-forge
- omnia
- defaults
- main
Perhaps a slightly better error message would be nice, considering the main channel is not added by default.
Here are the new and improved error messages for the tox failures:
E AssertionError: .
E ./.docker
E ./.docker/contexts
E ./.docker/contexts/meta
E ./eleventh
E ./fifth_writable_directory
E ./first_writable_file
E ./log.txt
E ./nineth_writable_directory_literal
E ./second_read_only_file
E ./sixth_read_only_directory
E ./twelfth
E /my_path
E /my_path/my_file_literal
E /my_path/seventh_writable_directory
E /my_path/tenth_writable_directory_literal
E /my_path/third_writable_file
E /my_other_path
E /my_other_path/eighth_read_only_directory
E /my_other_path/fourth_read_only_file
E .
E ./.docker
E ./.docker/contexts
E ./.docker/contexts/meta
E ./eleventh
E ./fifth_writable_directory
E ./fifth_writable_directory/c
E ./first_writable_file
E ./log.txt
E ./nineth_writable_directory_literal
E ./second_read_only_file
E ./sixth_read_only_directory
E ./twelfth
E /my_path
E /my_path/my_file_literal
E /my_path/seventh_writable_directory
E /my_path/seventh_writable_directory/d
E /my_path/tenth_writable_directory_literal
E /my_path/third_writable_file
E /my_other_path
E /my_other_path/eighth_read_only_directory
E /my_other_path/fourth_read_only_file
E
E assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131
tests/test_iwdr.py:136: AssertionError
E AssertionError: .
E ./.docker
E ./.docker/contexts
E ./.docker/contexts/meta
E ./eleventh
E ./fifth_writable_directory
E ./first_writable_file
E ./log.txt
E ./nineth_writable_directory_literal
E ./second_read_only_file
E ./sixth_read_only_directory
E ./twelfth
E /my_path
E /my_path/my_file_literal
E /my_path/seventh_writable_directory
E /my_path/tenth_writable_directory_literal
E /my_path/third_writable_file
E /my_other_path
E /my_other_path/eighth_read_only_directory
E /my_other_path/fourth_read_only_file
E .
E ./.docker
E ./.docker/contexts
E ./.docker/contexts/meta
E ./eleventh
E ./fifth_writable_directory
E ./fifth_writable_directory/c
E ./first_writable_file
E ./log.txt
E ./nineth_writable_directory_literal
E ./second_read_only_file
E ./sixth_read_only_directory
E ./twelfth
E /my_path
E /my_path/my_file_literal
E /my_path/seventh_writable_directory
E /my_path/seventh_writable_directory/d
E /my_path/tenth_writable_directory_literal
E /my_path/third_writable_file
E /my_other_path
E /my_other_path/eighth_read_only_directory
E /my_other_path/fourth_read_only_file
E
E assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131
tests/test_iwdr.py:247: AssertionError
E AssertionError: Should be 2 cidfiles, but got [PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220822073109-807385.cid'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220822073110-839889.cid'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker/contexts'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker/contexts/meta')]
E assert 5 == 2
tests/test_examples.py:1125: AssertionError
Looks like the .docker/ directory is messing thing up.
@jfennick Thanks for the information!
I can't find any documentation on why Docker for Desktop, Mac edition adds that directory (and how to turn that off), so I've just hacked around it for now.
This pull request fixes 6 alerts when merging 22cd86dfa1a2170645ac14bba404eb78b8246bee into 0e2ced53f8cd2652c4117af697fe2f5c6c94fcb7 - view on LGTM.com
fixed alerts:
- 6 for Unused import
Looks like we're just about done. With this change, both pytest and tox work on my machine.
diff --git a/tests/test_examples.py b/tests/test_examples.py
index b6a7edab..0516a6df 100644
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -1120,7 +1120,7 @@ def test_cid_file_dir(tmp_path: Path, factor: str) -> None:
stderr = re.sub(r"\s\s+", " ", stderr)
assert "completed success" in stderr
assert error_code == 0
- cidfiles = list(tmp_path.glob("**/*"))
+ cidfiles = list(tmp_path.glob("**/*.cid"))
cidfiles_count = len(cidfiles)
assert cidfiles_count == 2, f"Should be 2 cidfiles, but got {cidfiles}"
This pull request fixes 6 alerts when merging 4b5948123d81497ffbc0c0ed8d425cdc24d206d5 into 0e2ced53f8cd2652c4117af697fe2f5c6c94fcb7 - view on LGTM.com
fixed alerts:
- 6 for Unused import
pytest and tox are now both working on my machine. However, I'm still running into #1694
This pull request fixes 6 alerts when merging bbda77363dcaaa8c0dbc0da1ba2d4a3ae19d54a4 into 92c4073bf2e7965587d62abbf08914e0735e5419 - view on LGTM.com
fixed alerts:
- 6 for Unused import