Is there a way to have arrows to and from Clusters

[alexandrnikitin]

I want to have arrows from and to Clusters, e.g.

    ELB("lb") >> Cluster("cluster1")
    // OR
    Cluster("cluster1") >> Cluster("cluster2")
    // OR
    with Cluster("cluster1"):
        cluster1 = [EC2("node1"),
                    EC2("node2"),
                    EC2("node3")]

    with Cluster("cluster2"):
        cluster2 = [EC2("node1"),
                    EC2("node2"),
                    EC2("node3")]
    cluster1 >> cluster2

All of them throw an exception right now. Is this use case supported or planned? Is there a workaround?

[mingrammer]

Inter-cluster edge is not supported now. Cluster class is for just grouping the nodes now. Let me research if it is possible (I think it could be achieved lhead/ltail options of Graphviz)

And 3rd way is not possible because Python does not support shift operation between lists. To support this method, we should create a grouping purpose class like “NodeList” so that let someone connects the nodes between clusters.

For now, you can connect the nodes between clusters in this way: https://diagrams.mingrammer.com/docs/examples#clustered-web-services

[jbpratt]

I've played around this morning with diagrams and have ran into a few issues with arrow direction. @mingrammer please let me know if I am doing something wrong.

From the step function, it should go into the first X. From the very last X lambda, it should go to both the MoveToOrg lambda as well as the S3 bucket -> SNS.

I also haven't found an appealing way to represent StepFunctions map state feature / concurrency within it

    with Diagram("AcctCreation", show=False):
        begin = (
            SQS("KickoffQueue")
            >> Lambda("AcctCreate")
            >> StepFunctions("StepFunctions")
        )

        with Cluster("StateMchne"):
            with Cluster("Cncrrntx3.1"):
                lambdas = Lambda("X") >> Lambda("X")

            x = lambdas >> Lambda("X") >> Lambda("X")

            with Cluster("Cncrrntx3"):
                z = x >> Lambda("X") >> Lambda("X")

            z >> [S3("X") >> SNS("Notif"), Lambda("MoveToOrg")]

        begin >> lambdas

Really enjoying this package, great job! Excited to share this with the team.

[mingrammer]

@jbpratt78 >> operation returns last node. So the return value of Lambda("X") >> Lambda("X2") is Lambda("X2"), therefore begin >> lambdas will connect the begin to Lambda("X2").

In this case, nodes that directly connect to the preceding node should be declared separately. You can write like this:

with Diagram("AcctCreation", show=False):
    begin = (
        SQS("KickoffQueue")
        >> Lambda("AcctCreate")
        >> StepFunctions("StepFunctions")
    )

    with Cluster("StateMchne"):
        with Cluster("Cncrrntx3.1"):
            entry_lambda = Lambda("X")
            out_lambda = Lambda("X")
            entry_lambda >> out_lambda

        inter_lambdas = out_lambda >> Lambda("X") >> Lambda("X")

        with Cluster("Cncrrntx3"):
            post_lambdas = inter_lambdas >> Lambda("X") >> Lambda("X")

        store = S3("X")
        store >> SNS("Notif")
        post_lambdas >> [store, Lambda("MoveToOrg")]

    begin >> entry_lambda

[kontrafiktion]

This issue now seems to contain two different issues. I am getting back to the issue of arrows to "clusters".

Yes, graphviz can have arrows between groups/clusters, but it is kind of a hack: You declare an edge between two nodes and add an annotation that you want to start/end at the cluster (https://stackoverflow.com/questions/2012036/graphviz-how-to-connect-subgraphs)

If you would want to implement that in mingrammer, e.g.:

    lb = ELB("lb")
    services = Cluster("Services")
    with services:
        svc_group = [ECS("web1"),
                     ECS("web2"),
                     ECS("web3")]

    lb >> services

you need to get one node from the services cluster, so that the correct code could be generated

To get the node, you either have to implicitly retrieve any one node from the cluster – so the cluster must keep all node objects (currently the node is immediately destructured when adding it to the cluster, AFAICS).

Or the node must be a assigned to a variable (e.g. svc_group_node1 and then something like:

    lb >> svc_group_node1.cluster()

[jbpratt]

@kontrafiktion my apologies, this was due to my lack of understanding as I thought my cluster usage was related. Thank you for the help @mingrammer

[ringods]

I'm with @kontrafiktion on this one. I would also like to see arrows between clusters. But allow also for the more Pythonic way of writing this:

lb = ELB("lb")
with Cluster("Services") as services:
    ECS("web1")
    ECS("web2")
    ECS("web3")

    lb >> services

BTW, have a look at #34.

[IaroslavR]

@kontrafiktion @ringods Can't reproduce this trick. diagrams 0.6.3. Full code of script:

from diagrams import Cluster, Diagram
from diagrams.aws.compute import ECS
from diagrams.aws.network import ELB

with Diagram("Cluster link", show=False):
    lb = ELB("lb")
    with Cluster("Services") as services:
        ECS("web1")
        ECS("web2")
        ECS("web3")
        lb >> services

failed with

Traceback (most recent call last):
  File "/home/iaro/PycharmProjects/diagrams_test/tmp.py", line 12, in <module>
    lb >> services
  File "/home/iaro/miniconda3/envs/diagrams/lib/python3.7/site-packages/diagrams/__init__.py", line 326, in __rshift__
    return self.connect(other)
  File "/home/iaro/miniconda3/envs/diagrams/lib/python3.7/site-packages/diagrams/__init__.py", line 372, in connect
    self._diagram.connect(self, node, directed)
  File "/home/iaro/miniconda3/envs/diagrams/lib/python3.7/site-packages/diagrams/__init__.py", line 168, in connect
    self.dot.edge(node.hashid, node2.hashid, **attrs)
AttributeError: 'Cluster' object has no attribute 'hashid'

[ringods]

@IaroslavR my code snippet was a proposal of how I would like to use it. It doesn't work like that (yet).

[IaroslavR]

@ringods Got it

[Karreg]

I +1 this! When you have 2 clusters, each one accessing 2 other clusters with many components in it, it makes the diagram less readable.

Also, it prevents from linking clusters directly.

[juv]

Another use case: I wanted to have a Kafka cluster and a Zookeeper cluster with 5 nodes each. All of the Kafka nodes are connected with - and the Zookeeper nodes as well. Now, when trying to visualize that there is a connection between the Kafka and Zookeeper cluster, one can either connect the two Clusters together (preferred) or visualize the connection from each Kafka Node to each Zookeeper node, as all Kafka brokers could could theoretically connect to each Zookeeper node. The latter generates a mess on my end so the proposed solution in this issue would be a great feature.

[mingrammer]

I'll review this proposal in a week.

[patrykorwat]

👍 for the feature.

I work on a tool to automate AWS infrastructure detection and draw diagram of the detected infrastructure. Right now, there's a high number of duplicates caused by redundant connections to multiple subnets from EC2 instances, Lambda functions etc.

Having this feature in place, the diagrams would look much cleaner and easier to understand.

[jaywarfield]

I use a corresponding feature on PlantUML and would add my request here for this feature too. Great tool - thanks!

[kjh235]

My current work around : Make the cluster a list, then point the middle elements together. It is a little more defining if you want multiple elements to have arrows, or just the cluster as a single arrow.

with Cluster("cluster1"):
        cluster1 = [EC2("node1"),
                    EC2("node2"),
                    EC2("node3")]

    with Cluster("cluster2"):
        cluster2 = [EC2("node1"),
                    EC2("node2"),
                    EC2("node3")]
    cluster1[1] >> cluster2[1]

[clayms]

@alexandrnikitin

I want to have arrows from and to Clusters, e.g.

Below is a work-around:

from diagrams import Diagram, Cluster, Edge, Node

graph_attr = {
    "layout":"neato",
    }

scaling_clus_attr = {
    "bgcolor":"transparent",
    "pencolor":"blue",
    "penwidth":"4.0"
    }

with Diagram("\n\nNodes on Cluster boundary\nwith connecting arrow", show=False, graph_attr=graph_attr) as diag:
    with Cluster("cluster 1"):
        A_UpLf = Node("", shape="plaintext", pin="true", pos="0,4")
        A_LwRt = Node("", shape="plaintext", pin="true", pos="4,0")

    with Cluster("cluster 2", graph_attr=scaling_clus_attr):
        B_UpLf = Node("", shape="plaintext", pin="true", pos="8,3")
        B_LwRt = Node("", shape="plaintext", pin="true", pos="10,1")

    A_tail = Node("", shape="plaintext", pin="true", pos="4,2")
    B_head = Node("", shape="plaintext", pin="true", pos="8,2")

    A_tail >> B_head

diag

[Karreg]

Is it working with some components inside the cluster? I tried your code but only ended with an arrow, no square...

[clayms]

@Karreg did you change the layout engine to "neato" ?

graph_attr = {"layout":"neato", }

[clayms]

Here is another way. It only works with the 'dot' layout engine, which is the default for this library, but I explicitly specified it for clarity.

The main thing to notice is that this library automatically sets the name attribute of the Cluster to be "cluster_" + self.label . See https://github.com/mingrammer/diagrams/blob/dfd8e0a52c8c4d1c3ce95dc7161c23bb2eaf0acb/diagrams/init.py#L223 You must keep this in mind when referring to the Cluster name, in the ltail and lhead attributes of Edge.

Below is a replication of Figure 22 from https://www.graphviz.org/pdf/dotguide.pdf using this library. This is a good example of how the Edge's head and tail are truncated at the Cluster boundary.

from diagrams import Diagram, Cluster, Edge, Node

graph_attr = {
    "layout":"dot",
    "compound":"true",
    "splines":"spline",
    }

node_attr = {
    "shape":"ellipse", 
    "height":"0.8",
    "labelloc":"c"
}

with Diagram("\nCluster to Cluster Edges", show=False, direction="TB", graph_attr=graph_attr) as diag:
    with Cluster("0"):
        a = Node("a", **node_attr)
        b = Node("b", **node_attr)
        c = Node("c", **node_attr)
        d = Node("d", **node_attr)
    
    with Cluster("1"):
        e = Node("e", **node_attr)
        f = Node("f", **node_attr)
        g = Node("g", **node_attr)

    h = Node("h", **node_attr)

    a >> [b, c]
    [b, c] >> d
    e >> [f, g]
    
    b >> Edge(lhead="cluster_1") >> f
    c >> Edge(ltail="cluster_0", lhead="cluster_1") >> g
    c >> Edge(ltail="cluster_0") >> e
    d >> [e, h]

diag

[clayms]

To have a more meaningful Cluster label, but still keep the Cluster name simple, just add the Cluster keyword argument graph_attr dictionary, and then set the "label" key to the desired text value.

with Diagram("\nCluster to Cluster Edges", show=False, direction="TB", graph_attr=graph_attr) as diag:
    with Cluster("0", graph_attr={"label":"More meaningful label"}):
        a = Node("a", **node_attr)
        b = Node("b", **node_attr)
        c = Node("c", **node_attr)
        d = Node("d", **node_attr)
    
    with Cluster("1", graph_attr={"label":"This is a longer label"}):
        e = Node("e", **node_attr)
        f = Node("f", **node_attr)
        g = Node("g", **node_attr)

    h = Node("h", **node_attr)

    a >> [b, c]
    [b, c] >> d
    e >> [f, g]
    
    b >> Edge(lhead="cluster_1") >> f
    c >> Edge(ltail="cluster_0", lhead="cluster_1") >> g
    c >> Edge(ltail="cluster_0") >> e
    d >> [e, h]

diag

[geoffreywiseman]

Sorry, does workaround still work? I've tried this as an experiment and I'm not seeing the arrows stop at the cluster edges:

with Diagram("Cluster Arrows", show=False):
    with Cluster("1a"):
        pub1a = PublicSubnet("1b")
    with Cluster("1b"):
        pub1b = PublicSubnet("1a")

    tgw = TransitGateway("tgw")
    tgw >> Edge(lhead='cluster_1a') >> pub1a
    tgw >> Edge(lhead='cluster_1b') >> pub1b

[clayms]

@geoffreywiseman You need to include the graph_attr dictionary, and set the "compound" key to "true".

See https://graphviz.gitlab.io/doc/info/attrs.html#d:compound

compound : bool, default: false If true, allow edges between clusters. See lhead and ltail below. Valid for: Graphs. Note: dot only

Example:

from diagrams import Diagram, Cluster, Edge, Node
from diagrams.aws.network import PublicSubnet, TransitGateway

graph_attr = {
    "layout":"dot",
    "compound":"true",
    }

with Diagram("Cluster Arrows", show=False, graph_attr=graph_attr) as diag:
    with Cluster("1a"):
        pub1a = PublicSubnet("1b")
    with Cluster("1b"):
        pub1b = PublicSubnet("1a")
    
    tgw = TransitGateway("tgw")
    tgw >> Edge(lhead='cluster_1a') >> pub1a
    tgw >> Edge(lhead='cluster_1b') >> pub1b


diag

[geoffreywiseman]

Ah, yes, I tried reproducing most of what you had, but the compound part I guess I didn't try (or failed to see that it worked). So that helps but not if I want two-way arrows:

graph_attr = {
    "layout":"dot",
    "compound":"true",
    }
    
with Diagram("Cluster Arrows", show=False, direction='TB', graph_attr=graph_attr):
    with Cluster("vpc"):
        with Cluster("1a"):
            pub1a = PublicSubnet("pub")
            priv1a = PrivateSubnet("priv")
        with Cluster("1b"):
            pub1b = PublicSubnet("pub")
            priv1b = PrivateSubnet("priv")
    
    tgw = TransitGateway("tgw")
    tgw << Edge(lhead='cluster_1a') >> priv1a
    tgw >> Edge(lhead='cluster_1b') << pub1b

[clayms]

@geoffreywiseman You'll need a Cluster for the ltails as well. Set the Clusters graph_attr as below if you don't want to see it.

from diagrams import Diagram, Cluster, Edge, Node
from diagrams.aws.network import PublicSubnet, TransitGateway

graph_attr = {
    "layout":"dot",
    "compound":"true",
    }

tgw_clus_attr = {
    "label":"",
    "bgcolor":"transparent",
    "penwidth":"0.0"    
}

with Diagram("Cluster Arrows", show=False, direction="TB", graph_attr=graph_attr) as diag:
    with Cluster("vpc"):
        with Cluster("1a"):
            pub1a = PublicSubnet("1b")
        with Cluster("1b"):
            pub1b = PublicSubnet("1a")
    
    with Cluster("tgw", graph_attr=tgw_clus_attr):
        tgw = TransitGateway("tgw")
    
    tgw << Edge(ltail="cluster_tgw", lhead='cluster_1a', minlen="2") >> pub1a
    tgw << Edge(ltail="cluster_tgw", lhead='cluster_1b', minlen="2") >> pub1b

diag

[bkmeneguello]

#407 does that nativelly, please check. Nodes can behave as Clusters.

[geoffreywiseman]

#407 does that nativelly, please check. Nodes can behave as Clusters.

Yes, #407 definitely looks like it'll do what I want in what seems like a simpler way; looking forward to that making it into a release. ;)

[chadfurman]

This is technically supported in Graphviz, looks like there might already be a PR for this https://github.com/mingrammer/diagrams/pull/439 which replaces https://github.com/mingrammer/diagrams/pull/407

[deeplook]

@mingrammer What is the current state of this ticket?