reproman

reproman copied to clipboard

reproman run with datalad* orchestrators should check versions/extensions

2

We have tried to run some computation on an elderly NITRC AMI (ami-0a4cf63709e153111) with datalad-pair-run orchestrator and container job parameter. It tried, but didn't work and according to stderr it...

yarikoptic

- log some details at the DEBUG level (e.g. that it is about to start an instance on ami blah in zone bleh) - issue a warning if no credentials...

yarikoptic

I guess we would need only an identity id for that instance, and somehow establish ssh keypair to access it

yarikoptic

for different projects/resources users might need to use different aws credentials. ATM AFAIK (but I could be wrong) we support only a single `[aws]` section within `~/.config/reproman/reproman.cfg`. We should allow...

yarikoptic

run: failed due to non output directory was fetched (and "conflicted" upon extraction)

1

We ran ```shell (git)smaug:…eproNim_Workshop_2019_1018/ReproNim_Workshop_2019_1018-reproman[master]git $> reproman run --follow -r my-nitrc-large --orchestrator datalad-pair-run --input "*/sub-*/ses-*/anat/*T1w.nii.gz" --output "simple_workflow_output" --jp "container=containers/repronim-simple-workflow" ./code/simple_workflow/run_demo_workflow.sh -o simple_workflow_output -w workdir --plugin_args 'dict(n_procs=96)' */sub-*/ses-*/anat/*T1w.nii.gz ``` which failed with...

yarikoptic

how should we "orchestrate" cleanup

1

TL;DR question(s): - should I just `reproman run ... datalad drop` to lighten up the remote copy of the dataset? - how to remove entire dataset -- smth like `p="$PWD"...

yarikoptic

reproman's datalad-pair-run run record should probably store some "reproman run" construct?

1

ATM datalad run commit record, in the `cmd` field just records the job id, e.g. ` "cmd": "20190920-124832-7bf3",` a full example ```shell (git-annex)hopa:…im/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3[master]git-annex $> reproman run --follow --input 'data/bids/sub-{p[sub]}' -r...

yarikoptic

RF backend-parameters -> backends and add option to list recommended "templates"

1

per our discussion during meeting. Also ideally it should pick up a list of recommended ones from 1. updated from online (our repo) 2. from local system so users could...

yarikoptic

1. initial run failed to another issue (#442) 2. I have tried to rerun but even removing remotes etc in the dataset but it didn't help 3. only then I...

yarikoptic

retrace venv's path results in retracing VCS under which it is

1

```shell 2019-05-30 10:51:51,027 [INFO] Entering iteration #1 over Tracers 2019-05-30 10:51:52,060 [INFO] DebTracer: 0 envs with 1 other files remaining 2019-05-30 10:51:52,063 [INFO] RPMTracer: 0 envs with 1 other files...

yarikoptic