How to generate graph diagrams like those included in documentation?

[trickmeyer]

How can I generate images like the link below without writing the graphviz, etc. myself?

https://pythonhosted.org/graphkit/_images/example_graph.svg

[ankostis]

graph = compose(...)
graph.net.plot("workflow.png")

You may use one of those file extensions: .png .dot .jpg .jpeg .pdf .svg

Alternatively you may open a matplotlib window:

graph.net.plot(show=True)

[ankostis]

There is a much more direct way, edited on my original reply.

[andres-fr]

@ankostis Thanks for your answer! This more direct way triggers an exception in my case, due to the output stream expecting a string and getting a binary.

My fix was to set the output file mode to "wb" (permalink):

https://github.com/yahoo/graphkit/blob/e70718bbc7b394280c39c1fda381bcebd4c3de8d/graphkit/network.py#L426

As a workaround, your prior suggestion works like a charm (png, svg...):

with open("/tmp/test.svg", 'wb') as f:
    f.write(graph.net.plot().create_svg())

Note that this requires pydot

[ankostis]

thanks. Will update my PR :+1:

[andres-fr]

The existing function was also creating the file in cases where the extension isn't supported. I think that it is better to match the extension first, and open the file after, something like this works for me:

def plot_graph(g, out_path: str) -> None:
    """
    :param g: A ``graphkit`` graph as the ones output by the
      ``graphkit.compose`` function
    :param out_path: The absolute output path with extension. Supported
      extensions: ``[png, dot, jpg, jpeg, pdf, svg]``.

    Plots the computational graph using ``graphviz``. For that requires
    ``pydot`` to be installed.

    .. note::

      This definition is almost identical to the built-in one, but it uses
      ``wb`` mode instead of ``w`` to avoid exception. For more details see:
      https://github.com/yahoo/graphkit/issues/13#issuecomment-536342255
    """
    basename, ext = os.path.splitext(out_path)
    plt = g.net.plot()
    # check if extension is known
    if ext.lower() == ".png":
        obj = plt.create_png()
    elif ext.lower() == ".dot":
        obj = plt.to_string()
    elif ext.lower() in [".jpg", ".jpeg"]:
        obj = plt.create_jpeg()
    elif ext.lower() == ".pdf":
        obj = plt.create_pdf()
    elif ext.lower() == ".svg":
        obj = plt.create_svg()
    else:
        raise Exception("Unknown file format for saving graph: %s" % ext)
    # if extension matched, save created object
    with open(out_path, "wb") as f:
        f.write(obj)

Would PR myself but seems that I need credentials? In any case, this is my code, feel free to adapt at will. Cheers! Andres

[ankostis]

Would PR myself but seems that I need credentials?

For this project specifically? You just need to sign an (automated) CLA, that you mean?

[andres-fr]

Yes, I will look into that for the next time. In any case I see that you are working on that and it probably would be a bad idea to interfer. Thanks for your effort!

[ankostis]

Thank you for the guidance.

Btw, are you related to this project? Do you know if it's alive?

[andres-fr]

I am not related to Yahoo, just noticed this project some days ago and did some tests. Still awaiting for a reaction, but I am unsure that it's active (last commit was march 2018). All I know about the company are rumors and speculations from the internet.

I like the project because it is understandable, lightweight and seems a fair compromise between dynamic types and predefined interfaces, otherwise most people would be happy with the DL alternatives out there, which already provide lazy graph flow and automated backpropagation.

So far I'm happy but the issue I opened regarding graph merging invites to be cautious and double-check things. To that extent I will be happy to contribute as long as I find out things, but probably not as a maintainer. Maybe we should wait a little longer and see what happens.

Cheers

[ankostis]

...otherwise most people would be happy with the DL alternatives out there,..

What is DL?

Given this discussion (and the state of this project), i put a link here with my study of alternative DI-IoC python projects before deciding this one as an executor for functions modifying pandas dataframes (#15, WIP): https://docs.google.com/spreadsheets/d/1HPgtg2l6v3uDS81hLOcFOZxIBLCnHGrcFOh3pFRIDio/edit?usp=sharing

[andres-fr]

Amazing analyisis, thanks! I'm sorry, DL->deep learning.

I did a similar review of DL frameworks, you can check it here: https://github.com/andres-fr/dl-frameworks/blob/master/latex-presentation/dl-frameworks-presentation.pdf

Right now is for sure outdated, but maybe can be of some use still.

[ankostis]

[edit] Impressive study that you did!

Btw, you expressed exactly my feeling about this project, and i believe this extract should make it to the landing page of the project:

I like the project because <<it is understandable, lightweight and seems a fair compromise between dynamic types and predefined interfaces...>>

[ankostis]

With #19, pipeline.plot() now draws: -execution-steps

• inputs
• outputs
• solution values
• "optional" needs:

Legend

Nodes:

• circle: function
• oval: subgraph function
• inversed-house: given input
• house: asked output
• polygon: given both as input & asked as output (what?)
• square: intermediate data, neither given nor asked.
• red frame: delete-instruction, to free up memory.
• filled: data node has a value in solution, shown in tooltip.
• thick frame: function/data node visited.

Arrows

• solid black arrows: dependencies (target need source, sources provide target)
• dashed black arrows: optional needs
• green-dotted arrows: execution steps labeled in succession

[bt-]

@ankostis, thank you for sharing the your spreadsheet comparison DI-IoC projects. After looking at some of the packages in your spreadsheet, I'm guessing that DI is Dependency Injection, but what is IoC? Is there common functionality between the DI-IoC and DAG-executor packages?

I found this project because I am looking for a lightweight package that has a clear API for creating DAGs and generating a diagram of the DAG. It seems based on your work on this issue that you are also interested in the diagram feature. The functions I would be using in the DAG wrap very straightforward calculations not ML or DL models.

I encountered the same issue as andres-fr. It would be great to have your contributions merged. Has there been any communication from the creators of the package?

After writing the above, I found a maintained version- graphtik.

[andres-fr]

Hi Ben,

As far as I know IoC means "Inversion of Control": https://en.wikipedia.org/wiki/Inversion_of_control

Regarding the framework choice, I'm glad that I went for an option that identifies nodes based on Python references and not on strings, it is so much less headache in my opinion.

Best regards, Andres

[ankostis]

After writing the above, I found a maintained version- graphtik.

Indeed there is not much mention of that fork i maintain here, not to appear as if i'm hijack this project.

But i can tell you the fork has many & significant fixes and enhancements. You may glance over them through the changes.