cbrain icon indicating copy to clipboard operation
cbrain copied to clipboard

flag to copy input data to work folder

Open MontrealSergiy opened this issue 2 years ago • 7 comments

This to reduce corruption of input data and race conditions with input mutating scientific tools, see #1281 for description

MontrealSergiy avatar Jan 27 '23 23:01 MontrealSergiy

a test boutiques is available at https://github.com/MontrealSergiy/cbrain-plugins-test/blob/copydata/boutiques_descriptors/copy_flag_demo.json

MontrealSergiy avatar Jan 30 '23 01:01 MontrealSergiy

I don't really want a new attribute in the tool config structure, rather I want a simple implementation that just copies files when make_available is called with a new option:

def make_available(userfile, file_path, userfile_sub_path = nil, start_dir = nil, copy_files = false)

On the Boutiques descriptor side, all I want is support for:

    "custom": {
        "cbrain:full-copy-input-files": [ "inputid1", "inputid2", "inputid3" ],

prioux avatar Feb 02 '23 20:02 prioux

  • BTQ integrator to set copy_files

Note that I see several uses of make_available, besides the BTQ integrator e.g. such as boutiques_input_subdir_maker, bids_app_maker

Should I address them too? Would people continue to use it in modules ?

perhaps, it safer to pull property automatically, maybe yet another parameter copy_input_auto = true?

MontrealSergiy avatar Mar 15 '23 21:03 MontrealSergiy

Looks like rebase gone wrong still working

MontrealSergiy avatar Mar 17 '23 14:03 MontrealSergiy

Ok, changed as requested ( without modules or old style integrator)

MontrealSergiy avatar Mar 17 '23 15:03 MontrealSergiy

Please also look at supporting this option in the original Boutiques integrator. This code only supports the new integrator. If it's too complicated, let me know. We don't use the old integrator anymore, so maybe it's not worth it.

prioux avatar Apr 20 '23 15:04 prioux

rebased

MontrealSergiy avatar Nov 21 '23 02:11 MontrealSergiy