forwardProcess results in exponential amounts of work when nodes have multiple outputs connected to another node

[jezell]

return await Promise.all(Object.keys(node.outputs).map(async (key) => {
    const output = node.outputs[key];
    return await Promise.all(output.connections.map(async (c) => {
        const nextNode = (this.data as Data).nodes[c.node];
        await this.processNode(nextNode as EngineNode);
        await this.forwardProcess(nextNode);
    });
});

Consider the case where output.connections has the same node multiple times. processNode and forwardProcess will be called on that node multiple times. If in turn the next node has the same type of case this results in exponential amounts of promises in a wait state. Even if the nodes themselves do nothing in their worker, it doesn't take too large of a graph to start queuing up 10s of thousands of promise waits and lock the process while they are resolving.

One potential solution is to have a set of nodes on the engine that have already been forwarded and not forward if the item is already in the set. This seems to solve the problem on my end so far, but not sure if there are some potential pitfalls I'm missing. Is there a better way to handle this?

[Ni55aN]

Is there a real situation when a scheme will have thousands of nodes? Does this implementation work correctly with task-plugin and module-plugin?

[Flynamic]

This is an example where process takes one minute or more to complete, with just a fairly moderate amount of nodes. I suspect that this happens due to the exact reason described here, because #410 fixed it for me.

[rete-js[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.