Store compiled dependencies In _build by version

[jwdotjs]

Elixir and Erlang/OTP versions

erlang 25.2 elixir 1.14.3-otp-25

Operating system

MacOS - Ventura 13.1 - Apple M1 Max

Current behavior

As an engineer, when I switch between branches to provide peer review to other engineers, if dependencies changed across branches, I need to re-fetch and re-compile those dependencies. On larger projects this process can take 5-20 minutes, which can lead to slowdowns in pair programming and getting new pull requests tested, especially if I keep switching back to a local branch that I'm working on where dependencies have changed.

Reproduction steps:

1. Start on a project that is branched off main
2. On that branch change dependencies in mix.exs (either add dependencies, remove dependencies, or upgrade dependencies)
3. Run mix deps.get
4. Run iex -S mix phx.server to compile dependencies and the project
5. Run git checkout main on that project
6. Run iex -S mix phx.server and observe an error saying the lockfile doesn't match the compiled dependencies and to run mix deps.get
7. Run mix deps.get
8. Run iex -S mix phx.server and observe dependencies are re-compiled

Expected behavior

Here is a proposal for deps and _build that would be a breaking change to how the project is compiled and starts up, but would speed up development by caching the compiled versions of each dependency at the cost of more disk space (which is likely an acceptable tradeoff for most).

Sample filesystem structure:

project
  deps
     name_of_dependency
         1.13.0
         1.29.3
  _build
     lib
        name_of_dependency
           1.13.0
           1.29.3

The expected outcome of this change would be when I switch between a branch that has dependency changes and a branch where I already compiled dependencies with different versions, there is no time-cost associated with compiling dependencies when I switch between those two branches after they've been compiled once each

[ericmj]

For deps/ we can do this. But for _build/ individual dependency compilation artifacts cannot be stored by version since the version of dependency A can affect dependency B, if B calls A at compile time. Instead we would have to group all dependencies together like this:

  _build
     lib
        SOME_VERSION_1
          dep_a
          dep_b
        SOME_VERSION_2
          dep_a
          dep_b

I am not sure what version they should be grouped by though. Maybe the hash of the lock file?

[josevalim]

Good call @ericmj. Perhaps the solution is to have branched builds. Something like this:

_build/
  dev/
  dev-branch/

Every time a new branch is started, it copies from the canonical "_build/dev". However, compile-time config will be an issue. When you change a compile-time config, the dep is not automatically recompiled, so if we copy from _build/dev and there is a new configuration, compilation will fail telling you that you need to explicitly recompile that dependency. Maybe we could detect those but there is additional work.

Note we also already have MIX_TARGET which could be used for this purpose but I think introducing a custom option would be fine. So, roughly, I think we need this:

1. Add version to dependencies in deps/
2. Allow build_per_branch: "branch_name"
3. build_per_branch will copy from the non-branched name but it must consider compile-time configuration

[josevalim]

However, compile-time config will be an issue. When you change a compile-time config, the dep is not automatically recompiled, so if we copy from _build/dev and there is a new configuration

I am working on this part of the issue on a branch. EDIT: This has been merged.

[Nezteb]

I am working on this part of the issue on a branch.

@josevalim Is that branch public by chance? Or possibly already merged? I couldn't find any obvious branches related to this issue. 😄

[josevalim]

It is local but not really in a state to be shared yet, sorry. :(

[teamon]

I just wanted to highlight that adding a version to deps would make NODE_PATH a lot harder when importing js files from dependencies (e.g. in the context of phoenix)

[josevalim]

@teamon that's an excellent point. We could have symlinks, so deps/foo/current always point to the current version, but it is clear the behaviour would have to be opt-in.