any possibility of supporting by chunk operations on graphs

[alperyilmaz]

Supporting GraphFrame in Spark (#86) would be a nice touch but it would be also very nice if it's possible to analyze large graphs with tidygraph even if they don't fit memory by splitting into chunks and do per-chunk operations.

There's wonderful package disk.frame which allows data frames to be split into compressed random accessible (fst) chunks and then allows dplyr verbs to be used to process each chunk.

Probably it's very difficult to split a graph and process as chunks but there might be some workarounds or algoritms to do so which might be integrated with tidygraph.

[alperyilmaz]

this master thesis, titled "Large-Scale Social Network Analysis with the igraph Toolbox and Signal/Collect", talks about Signal/Collect framework and says: (taken from section 3.2)

The Python Client implements a graph library. The core is the SCGraph class which provides a subset of the interface of the igraph library. Providing the same method signatures, the user can use this class to build a graph and execute algorithms on it using the same methods as if she would use the igraph library. In contrast, the commands are executed transparently on a (possibly distributed) Signal/Collect server instance.

This might give some hope that igraph can be run in distributed fashion

[harryprince]

@alperyilmaz I proposed a similar issue.

https://github.com/neo4j-contrib/neo4j-spark-connector/issues/90

[alperyilmaz]

hopefully this gets attention..