poor build performance compared to host system toolchain

[nmattia]

We recently ported a repo to hermetic_cc_toolchain and have noticed a huge spike in build times on our CI. When trying to narrow it down, it looks like the compilation is just slower when using hermetic_cc_toolchain. In our repo, the builds using the hermetic_cc_toolchain are close to twice as long.

I've tried to reproduce this with this Dockerfile. In this case, registering toolchains from hermetic_cc_toolchain adds 30% to the critical path. The underlying toolchains are not exactly the same (I couldn't easily get the test lib jemalloc to build with clang-17, so the system toolchain is llvm-14 and AFAIU hermetic_cc_toolchain uses llvm-17).

Any idea what might cause the slowdown?