Add package level caching to `forc-pkg`

[kayagokalp]

I recently started thinking about adding a cache for packages to forc-pkg. Which basically will mean that we can omit re-compiling unchanged packages by default. We have module cache which prevents re-compilation of unchanged modules but forc does not need to call into compiler for unchanged packages at all. This means very easy compilation time wins for us until we have a fully fledged incremental compilation.

Option 1

We can cache all the packages by default. If the referenced packaged is not changed we can directly use from cached result without asking/requiring anything from the user. While this would be frictionless we might end-up storing a lot of bloat depending on the cached package size. A small case study is needed for deciding if this is feasible.

Option 2

Add a cache field to package declaration fields. Example:

[dependencies]
package_a = { path = "../package_a", cache = true }
package_b = { path = "../package_a", cache = false }

We would cache package_a but not package_b. By default we can make cache = false for unspecified values.

Option 3

Same as option 2 but we can make cache = true by default. Meaning that user can chose not to cache and save space by trading it off with compilation time.

Any thoughts? cc @FuelLabs/tooling

[JoshuaBatty]

Nice, this is a great idea. My vote goes to Option 1.

• It mirrors the Cargo approach, leveraging a target/output directory to store build artifacts. This approach is familiar to many developers.
• From a UX standpoint, it offers a cleaner experience. I believe we don't need granular control over module caching.
• Given that the number of packages in a project will likely be much fewer than in a typical Rust project, concerns about disk space usage seem minimal in comparison.
• If disk space ever becomes an issue, users can always run forc clean, analogous to Rust's workflow.

Those are just my thoughts. Eager to see what the rest of the team thinks!

[JoshuaBatty]

In the compiler we've already implemented in-memory incremental compilation by caching the program's AST in the query engine. This helps avoid recompiling unchanged code and improves response times but is only available to long running processes like the language server as it holds the Engines in memory.

The next step is to extend this incremental compilation to work across actual builds (like forc build and forc check). The idea is to save the AST to disk and on subsequent builds, check if it's still valid based on changes to the configuration. If the AST remains valid, we load it from disk to skip recompilation, otherwise we recompile as needed. This should significantly speed up build times for unchanged code.