couler

couler copied to clipboard

Documentation: Difference to other meta workflow engines

3

Couler is a meta workflow engine, but I know of at least [ZenML ](https://zenml.io/home) and [Kedro](https://kedro.readthedocs.io/en/stable/index.html) which are meta workflow engines as well. While the _self-presentation_ of these other two...

Make42

Hi I am trying out one of the example using argo workflow and I am getting k8s api exeception. Please find the logs: > `(venv) (base) saurav@saurav-mac pythonProject1 % python...

smetal1

Need additional clarification/examples around using set_dependencies+map

2

# Summary I'm confused on how to properly use dependencies. Let's say I have a workflow with 4 groups of steps (A, B, C, D) and each has multiple subtasks...

varunm22

Support Apache Airflow as a backend

7

This issue is for discussing adding Apache Airflow as a backend in Couler that's consistent with the design proposed in https://github.com/couler-proj/couler/pull/24. If you are interested in contributing, please comment and...

terrytangyuan

Revert "home brew" code to official Python client

6

# Summary Change custom code to use the official Python API # Use Cases It's difficult developing new features and onboarding new developers to the codebase, as the underlying structures...

moshewe

Implement `parallelism` for `template`s

1

# Summary The current implementation does not allow creating templates specifying the `parallelism` field as per the [Argo spec](https://github.com/argoproj/argo-workflows/blob/95ad561aec5ec360448267b09d8d2238c98012e0/examples/parallelism-nested.yaml#L38). # Use Cases Many possible use cases, my current one is...

moshewe

Explicit parameter passing between steps

1

# Summary Example: ``` out1 = couler.create_parameter_artifact(path="/mnt/test.txt") out2 = couler.create_parameter_artifact(path="/mnt/test2.txt") def producer(name): return couler.run_container( image="alpine:3.6", command=["sh", "-c", 'echo "test" > /mnt/test.txt'] , step_name=name, output=[out1, out2] ) def consumer(name): inputs =...

hcnt

Example for usage of secret

2

# Summary What change needs making? Example script on how to use created secret successfully. I have tried myself using the following code from [test](https://github.com/couler-proj/couler/blob/b363f59eaba85f2e8099d24c08934fa93645999c/couler/tests/secret_test.py#L31-L38) It creates a secret successfully...

nooraangelva

Long term plan and documentations

5

this is exciting and would love to build on top of it. For us, we have 5 data scientists authoring workflow on airflow and I'd like to move them to...

binarycrayon

# Summary Remove Go/protobuf related code. # Use Cases Simplify codebase. Python implementation calls k8s API directly anyways. --- **Message from the maintainers**: Impacted by this bug? Give it a...

moshewe