Path normalization for containerized builds

[zmc]

For compiler caching to work properly, paths to source files need to be able to match across builds. That means things like the ceph version, git commit id, etc. should not be present in any path components.

Containerized builds will help greatly with this, and are a work-in-progress: CentOS builds appear to properly get a 100% cache hit rate on rebuilds. Ubuntu builds, however, do not. See this build of the WIP pipeline, particularly how the CentOS branch takes 30min to compile while the Ubuntu branch takes 73min.

We need to investigate how to normalize paths for Ubuntu and Debian builds.

[deepssin]

I’ve been analyzing the jammy vs CentOS build time gap. sccache is being picked up and logs show it’s initialized, but still seeing no cache hits on repeat builds. I’ve already tried wrapping the compiler with sccache clang and exporting it before CMake, but it doesn’t seem to be taking effect . Suspecting either the CMake toolchain setup is overriding it or the compiler path isn’t being preserved.

[deepssin]

Considering the CentOS build, it also has a dynamic path like /ceph/rpmbuild/BUILD/ceph-20.0.0-161-ge5c1df06/. However, we are still observing shorter build times with CentOS. It's unclear if this is due to sccache hits or the toolchain.

The debug logs from this link indicate that sccache is set up correctly, but there are significantly more misses than hits.

The message parse_arguments: Cannot use preprocessor cache because of "-MD" frequently appears in the sccache debug logs. Overriding with -MMD might help, but it's uncertain if it will affect build rules.

I have initiated runs without including the SHA in the path from build-with-container.py and will provide updates on the results.

[deepssin]

Compared multiple ubuntu runs , few observations

• compiler flags looks exactly same for almost all the build files with same build
• '-MD' flag looks to be in inteferring with sccache and cause cache misses but same is the case with centos builds as well.
• Tried to override the -MD flag from CMakeLists.txt from the main and the submodules but its not getting overridden, its not even configured to use that as well. Not sure from where its getting picked up

[zmc]

The issue with Debian builds is fixed by these PRs: https://github.com/ceph/ceph/pull/63411 https://github.com/ceph/ceph-build/pull/2371