pydeps icon indicating copy to clipboard operation
pydeps copied to clipboard

Feature Request: Pydeps for pipelines

Open landoskape opened this issue 1 year ago • 2 comments

Pydeps is an excellent tool for visualizing dependencies. I wonder if it could be adapted to show the dynamic dependencies of a particular pipeline. For example, imagine a simple pipeline built with numpy and matplotlib that goes from raw data to visualization.

Here's an example:

import numpy as np
from numpy_processing import process0, process1, process2 # which all have their own dependencies
from matplotlib_visualization import prepare_vis, vis # also with their own dependencies

fname = 'some_file_name.npy'
data = np.load(fname)

output0 = process0(data)
output1 = process1(output0)
output2 = process2(output1)

vis_prms = prepare_vis(output2)
vis(output2, vis_prms)

We've defined a pipeline which uses these methods in the following order:

  1. np.load
  2. numpy_processing.process0
  3. numpy_processing.process1
  4. numpy_processing.process2
  5. prepare_vis
  6. vis

Wouldn't it be cool to see a pydeps graph where these methods are provided as input, are positioned on the left column of a graph moving downwards, and the dependencies are shown to the right pointing to each part of the pipeline?

If this is possible, please let me know. If there's interest and I could be of assistance in writing code to support it, let me know.

landoskape avatar Jul 02 '24 14:07 landoskape