giraph icon indicating copy to clipboard operation
giraph copied to clipboard

[PageRank] Make one-to-many messaging configurable

Open dlogothetis opened this issue 6 years ago • 2 comments

https://issues.apache.org/jira/browse/GIRAPH-1189

The PageRank implementation uses one-to-many messaging to reduce the amount of traffic sent. The corresponding message request (SendWorkerOneMessageToManyRequest) creates ByteArrayVertexIdMessages instances before adding the messages to the Message Store. When the number of messages is big, the creation of this instances can fail.

Adding an option here to enable/disable the use of one-to-many messaging.

Tests:

  • Run with dataset that would cause PageRank to fail because of large number of messages.

dlogothetis avatar Apr 25 '18 06:04 dlogothetis

Why do we have to create ByteArrayVertexIdMessages before adding to MessageStore, would it be possible to fix that instead and then we get benefit on all apps?

majakabiljo avatar Apr 25 '18 13:04 majakabiljo

A more general solution at https://github.com/apache/giraph/pull/72.

dlogothetis avatar Apr 26 '18 15:04 dlogothetis