graph-learn功能空缺点

[bookug]

请问现在graph-learn是否支持spark分布式训练？多机多GPU分布式训练？ 是否支持TransformerConv，KGCN等较新的算法？ egograph是否可以对接pyg，或者像subgraph那样通过GSL转换数据集从而支持pyg? 训练/预测时各个点是独立训练的么，除了需要共享参数？ 每个点维护自身的k跳邻居信息，这种做法是否会导致较大的内存开销？（是否有量化的统计数据） PS: graphscope中GLE就对应graph-learn，这两边有区别么？会实时同步？

[baoleai]

Hi, thanks for your interest. Graph-Learn started out by encapsulating GNN models in a dense EgoGraph fashion for large-scale graphs, and built models primarily based on TensorFlow 1.12. With the popularity of frameworks such as PyTorch and PyG, we gradually started to provide support for PyTorch and also abstract GNN models to a SubGraph format similar to Data in PyG.

Currently Graph-Learn has relatively poor performance in multi-machine multi-GPU, and we are further optimizing our support for PyTorch multi-machine multi-GPU distribution, including GPU sampling, efficient feature queries, etc.

The EgoGraph format has a lot of data redundancy, so we are not considering continued support on PyTorch, and we intend to reuse the model part of PyG, etc. as much as possible.

Some new algorithms, such as TransformerConv, KGCN, etc., as you mentioned, can be implemented based on frameworks such as Graph-Learn, TensorFlow, and PyTorch, (https://graph-learn.readthedocs.io/en/latest/algo/tf/tf_custom_model_tutorial.html#customizing-the-model, https://graph-learn.readthedocs.io/en/latest/algo/torch/torch_custom_model_tutorial.html) and we welcome your contributions.

The deployment of Graph-Learn is independent of the specific framework, and Spark-based distributed training may require modifications to the file system (https://graph-learn.readthedocs.io/en/latest/other_source.html#file-system-interface) .

[bookug]

比如egograph，采样后是如何存储和使用的。分布式中图分割是分割原图而不是采样后的？ agl论文中是把每个点的跳邻居直接提取后单独存储（冗余很多，存储空间占用大），此后各个点独立训练，独立加载。 看起来graph learn的egograph虽然也是采样k跳子图，但应该和agl做法不同？

[baoleai]

Yes, we partition and store the original graph in a distributed manner, and then sample and train from the original graph. EgoGraph is just the organization form of the sampling result, not the storage form of the graph. EgoGraph will also have some redundancy, because it is simply composed of neighbors layer by layer, without deduplication.