pepkit
Optimize how and when looper checks pipestat config and allow user to overwrite when running samples
Originally, looper would overwrite the pipestat config upon executing looper run. However, several months ago, we changed it such that it will not overwrite the pipestat config if it finds one already in place.
Running my own pipelines, I've found this behavior confusing and annoying as its led to me changing items in my looper config and then not seeing the changes executed downstream in results reporting. I then must hunt for the location of the pipestat config and delete it before re-running.
I'm contemplating adding the ability to preserve the previous config and create a new one for the current run. This would give us preservation of previous settings for debugging etc and remove the above friction when using looper + pipestat.
Or I may return to having the default behavior be to overwrite the pipestat config. Or perhaps have a flag to toggle such behavior.
This just bit me again this morning and caused a collision of reporting results to the wrong PEP this morning. Related to https://github.com/pepkit/pephubclient/issues/53
It saves time to assume you want to use the existing config instead of creating one over and over again. I will leave it as is and close this issue.
I went to investigate and remind myself why it was faster and found that looper checks if pipestat is configured as it adds each sample to the Conductor. So, there was the potential to write a new config each time it processes a sample.
I think there are two steps to correctly solving this:
- [ ] change how looper checks for pipestat configuration (right now its checking initially during piface processing and then when adding samples).
- [ ] allow user to overwrite or use existing pipestat config (we use a pipestat manager for each piface) Set default to overwrite.
Reopening to track this work and I'll edit the title.