spring-data-neo4j icon indicating copy to clipboard operation
spring-data-neo4j copied to clipboard

SimpleNeo4jRepository.findAll(Pageable) fails on big databases

Open bonelli opened this issue 3 years ago • 0 comments

Invoking SimpleNeo4jRepository.findAll(Pageable) will cause OOME on databases with many nodes, even with a small page size.

After some debugging, I found the probable cause in these lines of the Neo4jTemplate.createNodesAndRelationshipsByIdStatementProvider() method: https://github.com/spring-projects/spring-data-neo4j/blob/402c0bacb4f9b0f70091967d39ef4fdb95437cec/src/main/java/org/springframework/data/neo4j/core/Neo4jTemplate.java#L1142-L1144

The rootNodesStatement that is used to detect circular dependencies is quering the database for all nodes of the given type, extracting their ID.

This is no big deal on small databases, but in my case I have a single database with +10M nodes of the given type (e.g. Foo nodes), and the JVM starts downloading many millions of IDs from the database, eventually crashing for OOME or failing the execution for query timeout.

Is it essential to request the complete set of IDs for all of the Foo entities in the database, even when I'm just trying to fetch one page with 20sh elements?

bonelli avatar Sep 19 '22 14:09 bonelli