Don't reclone and rebuild duplicate dependencies

[tionis]

Currently, when I have duplicate dependencies in my recursive dependency graph they are built multiple times. I suggest building the dependency tree first, and then maybe apply the algorithm in jpm/dag to build each only exactly once. Example:

• Project A deps: [B, spork, C]
• Project B deps: [spork]
• Project C deps: [B] which results in the following graph:

graph
    a-->b
    a-->spork
    a-->c
    b-->spork
    c-->b

Which means the most optimal build process would be to (0. fill cache by walking the graph and cloning new deps as needed)

1. build spork
2. build b
3. build c
4. build a

But currently it looks more like this:

1. clone & build spork
2. clone & build B
3. clone & build spork
4. clone & build spork
5. clone & build B
6. clone & build C
7. clone & build A

[sogaiu]

I agree it would be nice for the duplicate building to be reduced / eliminated.

---

Nice graph!

[bakpakin]

If the build system is doing it's job, this should really cause things to slow down very much since things shouldn't be rebuilt from scratch. Does this actually slow things down that much?

JPM lockfiles aim to fix this issue by installing all unique dependencies only once, in an order where all of a libraries dependencies are installed before it is installed.

[sogaiu]

@tionis Did you have a chance to try the new code from #85?

[tionis]

No, the two issues are currently not on top of my todo list so I haven't found time yet.

[tionis]

But I think the issue is less severe. Have to test some more in a clean env

[tionis]

I think it's mostly fixed. jpm still builds dependencies ofter than needed, but that only results in the files being copied into the package cache (i'm unsure on the terminology here) more than needed. Compilation and similar tasks are not duplicated anymore.

[tionis]

I'm unsure if I should close this, as the issue is technically not fixed.

[sogaiu]

Thanks for checking!

May be leaving it open is ok for now.