Large graphs: Faster graph object creation

[shadiakiki1986]

Problem

• I would like it if large graphs (eg 5k nodes, 30k edges) loaded faster than 2 minutes

The slowest part for me is the edge creation. The Edge() constructor is called n-edge times at cytoscape.py#L524. It's not the Edge constructor that's slow, but the Widget constructor from which Edge inherits.

Proposed Solution

I already started a SO question asking about the most efficient way to create a large number of Widget objects here. It seems to me that the solution would require a new EdgeList class that internally handles a list of edges and just needs a single Widget constructor call, but I don't know the internals, so this might as well be non-sense.

Btw, the same applies to the Node constructor, but it's a smaller problem for me ATM

[marimeireles]

Thanks for opening the issue!

It seems to me that the solution would require a new EdgeList class that internally handles a list of edges and just needs a single Widget constructor call, but I don't know the internals, so this might as well be non-sense.

This makes sense to me, it seems like it'd work... I'm just worried with the synchronization problems that spectate have with deep objs.

@MridulS implemented most of these methods :) You have an opinion on this?

[MridulS]

It seems to me that the solution would require a new EdgeList class that internally handles a list of edges and just needs a single Widget constructor call, but I don't know the internals, so this might as well be non-sense.

This makes sense to me, it seems like it'd work... I'm just worried with the synchronization problems that spectate have with deep objs.

This should be possible but I think this will take up some initial effort to implement the EdgeList class and keep the syncing, like this will probably mean that once the graph is created you can't delete any edge manually from that EdgeList (or recreate the EdgeList class with N - 1 edges) which (IMO) kind of breaks functionality as it's nice to have an interactive graph where you can remove and add nodes/edges as required. It's possible there's a better solution, I am not an expert in jupyter widgets :)

[marimeireles]

Yes, that would be terrible. But (I just look at the code)...

edges = MutableList(Instance(Edge)).tag(sync=True, **widget_serialization)

Theoretically, the interactivity should be preserved because edges is a MutableObject already. So, I'd say this idea sounds good. :) If you're willing to tackle it, please go ahead, I'm around here and on the https://gitter.im/QuantStack/Lobby channel if you have any doubts.