argo-workflows
argo-workflows copied to clipboard
argoproj
Consider implementing the Common Workflow Language standard
maybe we could consider to parse both wdl/cwl into DAG and parse inputs as parameters in argo, take NAS volume as an example PoC, then go object storage.
https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.wdl https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.hg38.inputs.json
This is unlikely to be developed by the core team at this time - it is a perfect piece of work for community contribution.
Picked this up, going to be developing on a CWL to argo transpiler on my fork https://github.com/isubasinghe/argo-workflows
Hey @mr-c would you by any chance have time for a quick chat about CWL? Perhaps we could setup a small zoom meeting?
Thanks I have booked a meeting for Monday morning, looking forward to chatting with you
Apologies for not getting back on to this, my latest work is available on my branch here: https://github.com/isubasinghe/argo-workflows/tree/feat/cwl2argo-output/cmd/cwl2argo
It is capable of compiling "CommandLineTool" types but it would be trivial to reuse the functions to make the compiler work for "Workflow" types.
Unfortunately I am busy with university and settling in another country at the moment so I haven't had time to work on Argo Workflows.
@alexec Hi, I am happy to pick this up, it's come up at work a couple of times over the last year.
Would you consider a PR where this is encapsulated as a new argo cli command? Something like, argo transpile FILE ?
Hey @SerRichard, I'd be happy to help you out with this. I have a parser for CWL that would help you. It already supports some amount of argo.
It's unfortunately not as simple as I personally thought it was at the time.
@isubasinghe The code in the fork, or something else? & do you remember the sticking points you found when trying to work on it before?
For parsing CWL in Go, I recommend adding Go support to our code generator. It is already used to make CWL parsers for C++, Python, Java, Typescript, .Net, and soon also Rust:
https://schema-salad.readthedocs.io/en/latest/#codegen-examples
(I can provide guidance and mentoring for this task)
@SerRichard Yeah the fork has an additional cli, I think one of the branches are pretty much complete for a CWL type as well.
Probably not much for you to do from a parser and type checking point of view. I can't remember the exact pain points, but generally speaking the correspondence between CWL and argo isn't one to one and requires some creativity to address.
My tool already does some generation, this: https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/argo_simple.yaml was generated from https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/locations.json and https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/hello.cwl
@isubasinghe @mr-c Sorry for the delay replying, I've only now had a chance to look at this again. Going through the cwl spec there's a couple of places where I'm not certain what the expected behaviour of a cwl2argo parser would be.
Firstly, take the example of a CommandLineTool where no docker requirement is defined, or only software requirements are defined. Install commands could be added to a step's script in a workflow, but in both cases I would still expect the user to define some base environment that this cli is expected to run in. So could Docker requirements with a docker pull be a requirement for the parser? If not, do you have any other ideas on how to handle this?
Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.
Any input is appreciated!
So could Docker requirements with a docker pull be a requirement for the parser?
Yes, that is reasonable, and true for basically all the cloud-based CWL compatible platforms 👍
Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.
I'm not clear on the question here. Can you share some examples?
Hi again, sorry for the delay! On my fork I've taken the initial implementation from @isubasinghe and extended to also work for the CWL Workflow, currently I'm still missing some handling ( e.g. scatter & scatter method, and other bits ), I need to write a set of unit tests, and I'll need to tidy up the branch quite a bit, but the basic parsing for a CWL Workflow is there.
@alexec I'd like to open a draft PR, in order to get some feedback and continue the work with some general direction.
@mr-c forget my earlier question, I was a bit muddled about when to use an artefact vs output paramter when transpiling to an argo workflow. Though, I think that this can either be set explicitly, or inferred from the class type of the CWL provided. Additionally, the type checking has grown quite a lot, and will continue to grow if I were to add support for Operations and ExpressionTools. I might be able to work on your earlier mention of Go support for parsing CWL, if you like we can set up a meeting and discuss that in parrallel?
Curious, is there any reason for this tool to exist as part of the core Argo repo? It sounds like it would work perfectly fine as its own separate tool, no?
We can certainly host it in argoproj-labs as well
@mr-c I might be able to work on your earlier mention of Go support for parsing CWL, if you like we can set up a meeting and discuss that in parrallel?
Happily! My email username is mrc and the domain name is commonwl.org
Here's a video from this year's CWL conference showing how they did the same for Rust: https://cwl.discourse.group/t/salad-goes-rusty-expanding-the-horizons-of-cwl/907
@agilgur5 I was thinking the same while I was working on this ( albeit not specifically where the tool would live ), though @isubasinghe might know of a reason why this was planned to be included in the core repo? I'm happy to continue the work as part of a tool in the argoproj-labs project, for my own use cases this might even be preferable.
@SerRichard no real reason that it should exist in this repo. It would be better served in argoproj-labs.
It was early days into my argo workflows contribution that I worked on this and just didn't know about the existence of argoproj-labs.
Can we support WDL too? We have encountered many wdl workflows running on cromwell that want to run on kubernetes.
