cargo xbuild doesn't take advantage of CARGO_TARGET_DIR
I'm not sure exactly how this plays out, but I compile a number of crates using cargo xbuild, and I try to set CARGO_TARGET_DIR so that they all share a target/ directory and I only need to build the various dependencies once. However, core, compiler_builtins, and alloc seem to get rebuilt every time.
You can see this in the travis output for any build on https://travis-ci.org/alex/linux-kernel-module-rust
You could try setting the XBUILD_SYSROOT_PATH environment variable.
I agree that it might make sense to consider the CARGO_TARGET_DIR environment variable too for placing the sysroot. However, there are some problems with using the same sysroot for mulitple independent projects due to some internal checksums, if I remember correctly.