python-pachyderm icon indicating copy to clipboard operation
python-pachyderm copied to clipboard

Support python function pipelines

Open lukemarsden opened this issue 5 years ago • 1 comments

Enable a data scientist to define pachyderm pipeline steps as python functions (e.g. from a jupyter notebook).

lukemarsden avatar Aug 11 '20 14:08 lukemarsden

To add a bit of color:

  • I think a python library could use getsource() to get the body of a python function/object, and then commit that to Pachyderm and run it in a standard python container. I believe kubeflow does this for function-based components.
  • Deps could be downloaded and committed to pachyderm as well)

msteffen avatar Jun 16 '21 05:06 msteffen