redisgraph-bulk-loader icon indicating copy to clipboard operation
redisgraph-bulk-loader copied to clipboard

How scalable is this?

Open uriva opened this issue 5 years ago • 1 comments

  1. Would it support a chunk of 5 billion nodes/edges?
  2. If each has minimal payload, how much time would the process take?

uriva avatar Aug 07 '20 12:08 uriva

Hi @uriva,

  1. If your server has enough RAM to store and query a graph with 5 billion entities, you should not have an issue running the bulk loader. It will automatically divide your input into batches to populate a buffer of up to 2 gigabytes, and maintains a dictionary mapping all nodes to their identifiers.
  2. I'd expect this to take dozens of hours, but there are too many factors in play to be very precise. Generally, load time will scale linearly with the input size. Building a graph with about 5 million nodes, 5 million edges, and 20 million properties on my system takes 220 seconds, so increasing that by a factor of 500 gives about 30 hours as a very very rough estimate.

jeffreylovitz avatar Aug 10 '20 16:08 jeffreylovitz