looper
looper copied to clipboard
pepkit
Feature request: Skip pipeline location check
For running a pipeline within a container there is a need to skip the check for pipeline location verification as the pipeline of interest in hidden from looper's view within a container.
This refers to looper's "Missing pipeline script: ..." and "reasons for submission failure: No pipeline for protocol." messages in this scenario.
If the pipeline lives inside a container, the pipeline interface will still have to live outside the pipeline, because looper needs direct access to the pipeline interface. Users must therefore clone the pipeline locally anyway (or at least have a pipeline interface), even if using a containerized version.
Then, how should a user instruct looper to use a containerized pipeline, as opposed to the regular one? One solution is two pipeline interfaces -- one that indicates a containerized version, the other a local version. But let's not do two pipeline interfaces because it duplicates stuff.
For the containerized version, looper will need 2 things:
- container name
- location of pipeline within the container
These could just be added in a 'container' section of the existing pipeline interface.
But how does the user tell looper to run in a container vs normal? Should it be on the command line (--container=blah) or within the project_config.yaml ?
was just thinking... 2 ideas:
-
Can't we solve the 'knowing the location of the pipeline within container' by using an ENTRYPOINT? Then we just execute the container and it will automatically execute the pipeline. the author would encode that info in the dockerfile. does singularity honor dockerfile entrypoints?
-
Can we avoid this problem altogether by not pulling the pipeline inside the container, but still running the whole pipeline (externally mounted) in the container? In other words, we don't do the original job-by-job method, it's still the whole pipeline in one run, but that pipeline actually lives external to the container. It is mounted into the container at runtime, and then executed within the container.
this strategy has a few advantages:
- upkeep on the image is much easier, because it doesn't have to contain the pipeline; so pipeline updates are as simple as a git pull
- one image could in theory support multiple pipelines quite easily
- the pipeline itself would still live outside the container, solving the 'missing pipeline script' issue -- because the pipeline interface naturally lives outside the container, too.
- we also no longer have to worry about telling the image where the pipeline is, because it's not there; it's local.
As discussed, I will add a way to disable the pipeline check as a simple solution for this problem.
It appears that this check is no longer happening. Looper will happily submit a pipeline that does not exist without issue.
Do you want me to reinstate this check and allow it to be toggled via the looper config file?