Performance at the scale of thousands of nodes

[dwelch2344]

Just came across Rete and am blown away. Seriously, great work! It's solving a few problems and ideas I have, especially with regards to teaching stream processing to junior engineers (and getting my kid coding!) so thanks and kudos to all involved.

I'm now trying to use Rete as my new "hammer for all my nails" and found a few great use cases. One of which I'd like to explore is jungling some graph data I have. Initial samples and workflows are working great, but I tried wiring it up to a few thousand nodes (3,266 in an organization hierarchy to be exact) and immediately felt the pain.

There's no real data in the nodes, they're using the default renderers (though it is initialized in a vue app), and I don't have any workers / engine even setup in the process – so I'm wondering if it's just the number of nodes and connections is getting burly for a browser in general.

I fished through the issues tonight and found the profiler plugin just barely, so I'll give that a crack tomorrow. But, in the meantime, figured I'd ask here to see what experience others have had with large graphs (even in just the editor, not the engine)

Thanks!

[jezell]

Could be related to this?

https://github.com/retejs/rete/issues/402

Have a fix locally that I can push up to a fork if you are having similar issue.

[dwelch2344]

I saw that! Might be. Nodes are definitely sporting multiple connections. Would love to take a look / help however possible

[jezell]

Try this: https://github.com/retejs/rete/pull/410

[codynova]

@dwelch2344 Any chance you can provide a sample data set and/or project? I'm working on some performance improvements to the engine and would love to have a solid test case.

[jezell]

Looks like the recursion checks also suffer from same problem before execute. causing some of our processes to take seconds to load, removing the recursion check makes them load instantly. Should be able to figure out a way to improve the perf of the check, or at least let it be optional at runtime.

[rete-js[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.