python-pachyderm
python-pachyderm copied to clipboard
Support python function pipelines
Enable a data scientist to define pachyderm pipeline steps as python functions (e.g. from a jupyter notebook).
To add a bit of color:
- I think a python library could use
getsource()to get the body of a python function/object, and then commit that to Pachyderm and run it in a standard python container. I believe kubeflow does this for function-based components. - Deps could be downloaded and committed to pachyderm as well)