Cargo build caching/incremental build support

[david-mcgillicuddy-moixa]

Version(s) of meta-rust

Master 11aed43

Version(s) of poky and/or oe-core

Rocko

Expected result

If two commits have only a single file difference, compilation should (could?) be very quick.

Actual result

If two commits have only a single file difference, it's still a full clean build each time

Steps to reproduce

N/A

I mostly want to discuss the possibility of enabling builds to use cargo artifacts of previous builds. In effect ignoring bitbake's Shared State Cache in favour of Cargo's cache. Currently it seems that as soon as the commit to the rust code is changed then everything is invalidated. Our project's Rust binary currently takes ~20 minutes to compile from scratch but only a few seconds from an incremental compile.

Is this desired? Feasable? I don't really know how it could be implemented (something to do with faking hashes for bitbake?) but am happy to give it a go with a few pointers. After all, I currently have a lot of time while waiting for builds ;)

[rwmacleod]

Ideally both the sstate-cache and the cargo cache could be used but I'm not sure if that's possible. The C/C++ equivalent is would be ccache (compiler-cache) / icecc/icecream + sstate working together.

[tylerwhall]

As long as we're using Cargo to execute the entire build plan within one recipe, sstate can't cache anything but the compiler, sysroot, and cargo-native.

https://github.com/mozilla/sccache might be worth a look. Looks like it might have some correctness issues due to not hashing all possible variables that affect the output, but maybe it would be good enough for local builds if you're not switching machines often.

I don't know off-hand what side effects of invalidating do_fetch cause Cargo to do a full build. Could be the time stamps of the extracted crates, and/or re-creating ${CARGO_HOME}/config in cargo_common_do_configure, and/or a directory in WORKDIR getting cleaned. If we know the cause, we might be able to work around it and allow "dirty" builds even when some sources change. Could start by bisecting tasks that come before do_compile (e.g. bitbake -C configure foo) so see which ones trigger a full rebuild. This only applies to local builds, though.

[janderholm]

If you use "devtool modify" to develop the package the cargo cache is used between compilations. It will however need to rebuild from scratch when you update the SRCREV resulting in long build times. But that is how many packages (e.g. the kernel) work when you update them.

Also setting DEBUG_BUILD="1" in the recipe (or DEBUG_BUILD_pn-packagename = "" in local.conf) will make bitbake compile in debug mode. The binaries tend to be very large though.

You can also set up a shared sstate cache between multiple products. So if you use devtool {modify, build, deploy-target} for local development and push it to a build server once the code is ready for use you can set up a job that syncs the sstate cache to a shared place. That way no one else needs to wait for a full 20 minutes unless they actually need to modify the code.