saving of hidden files that are not updated with changes to heuristic

[poldrack]

hi - it appears that information is saved into the .heudiconv folder that will prevent the heuristic from being rerun on the same data, even if the heuristic file changes. I spent several hours trying to debug a problem before realizing this. saving info to a hidden folder seems like a suboptimal solution, but if you are going to do it, please clear it out whenever the heuristic changes, or at least save it within the newly generated subject directory so that when that directory is deleted (which is necessary in order to rerun the job) then the hidden directory would be deleted as well.

[satra]

noted - we did a big merge with a large number of changes. time to clean up the changes from this.

[yarikoptic]

This I think bitten me as well a few times. But I just found peace of mind with it as a feature ;-)

[satra]

@poldrack - as we move forward with this codebase, we will be bringing in a few more things.

• change in heuristic | heudiconv | environment will trigger a regeneration. at present there are two primary sources of difference: heuristic, dcm2niix but changes could come from heudiconv itself as well.
• integration of dcmstack into nibabel and/or extraction of features via other means

i think the biggest drawback from a testing standpoint is the absence of sufficient varied dicom datasets.

[yarikoptic]

I think that the problem is also due to trying to address two target use-cases:

• original two stage:
  1. generate .edit.txt with mapping to be adjusted if needed
  2. run actual conversion
• single stage:
  1. convert what is given without giving even a chance to adjust anything

The "single stage" is kinda my use-case which I added later, while "original two stage" is what Satra+Michael envisioned/implemented originally. And ATM it is not clear when heudiconv performs in two stage or a single stage scenario.
In my case, I am trying to avoid silent 're-conversion' since it could lead to spurious files laying around left from previous attempts with previous heuristic and inconsistent conversion altogether. That is why added explicit --overwrite flag to instruct when I explicitly want to overwrite previous results of the conversion, and otherwise to crash if any files tries to override any previous present file (be it a converted or a heuristic file).

I think we should sit down (during code rodeo) and outline major use-cases/workflows we would like to see supported, and make them clear(er) presented in API/documentation.

[satra]

@yarikoptic - sounds like a good discussion topic for code rodeo.

[yarikoptic]

I wonder if we should make --overwrite from a flag into nargs='*'option, thus allowing for no value which would default to 'converted-files', and have other values allowed such as 'mapping-tables' and 'all' for both.