[3/5] support C++20 Modules, add deps-scanner and update toolchains

[PikachuHyA]

Summary

I have splited the XXL PR #19940 into several smaller patches. This is the third patch to support C++20 Modules, which adds the deps-scanner tool and updates toolchains.

This patch includes:

1. New action names
2. File extensions
3. Build variables
4. Updated toolchains for compiling C++20 Modules

Action Names

Three action names have been added:

• c++-module-deps-scanning
• c++20-module-compile
• c++20-module-codegen

When two-phase compilation is employed:

• c++-module-deps-scanning: Scans source files and retrieves C++20 Modules dependencies, storing them in <filename>.ddi.
• c++20-module-compile: Compiles the C++20 Modules Interfaces to a Built Module Interface (BMI), converting <filename>.cppm to <filename>.pcm.
• c++20-module-codegen: Compiles the BMI to an object file, converting <filename>.pcm to <filename>.o.

When one-phase compilation is employed:

• c++-module-deps-scanning: Operates similarly to two-phase compilation.
• c++20-module-compile: Compiles the C++20 Modules Interfaces directly to an object file <filename>.o and produces a BMI <filename>.pcm as a byproduct.

File Extensions

We follow the file extensions preferred by different compilers, adding two new ArtifactCategorys: CPP_MODULE_GCM and CPP_MODULE_IFC.

• Clang uses .pcm (CPP_MODULE, already exists).
• GCC uses .gcm (CPP_MODULE_GCM, new).
• MSVC uses .ifc (CPP_MODULE_IFC, new).

Following the CMake implementation, we added three extra ArtifactCategorys: CPP_MODULES_INFO, CPP_MODULES_DDI, and CPP_MODULES_MODMAP.

• The .ddi file (CPP_MODULES_DDI) stores the dependencies information of one source file.
• The .CXXModules.json file (CPP_MODULES_INFO) stores dependencies information for an entire target.
• The .modmap file (CPP_MODULES_MODMAP) maps module names to BMIs, with different formats for each compiler.

Additionally, a special ArtifactCategory, CPP_MODULES_MODMAP_INPUT, is an auxiliary file used to easily obtain the requested BMI paths.

Build Variables

Two build variables, CPP_MODULE_MODMAP_FILE and CPP_MODULE_OUTPUT_FILE, have been added.

• CPP_MODULE_MODMAP_FILE specifies the path to the .modmap file and is used by the cpp20_modmap_file_feature.
• CPP_MODULE_OUTPUT_FILE specifies the output name of the BMI when one-phase compilation is employed and is used by the cpp20_module_compile_flags_feature.

Toolchains

Three action configs (cpp_module_scan_deps, cpp20_module_compile, and cpp20_module_codegen) have been added, corresponding to the action names section.

Two features (cpp_module_modmap_file_feature and cpp20_module_compile_flags_feature) have been added, corresponding to the build variables section.

Using C++20 Modules necessitates topological ordering for the compilation units. For more details, see the Discovering Dependencies section.

Considering the various compilers, I have added the deps-scanner tool. The default implementation is a script wrapper that uses different scanning methods depending on the compiler. The wrapper deps_scanner_wrapper is generated by a template file <compiler>_deps_scanner_wrapper.sh.tpl. Three template files have been added:

• clang_deps_scanner_wrapper.sh.tpl
• gcc_deps_scanner_wrapper.sh.tpl
• mvsc_deps_scanner_wrapper.bat.tpl

For a demonstration of how to scan C++20 dependencies, please refer to this demo.

[PikachuHyA]

I just keep the main commit in this patch. due to patch dependencies, it may build failed.

I also fix clang_deps_scanner_wrapper.sh.tpl according to https://github.com/bazelbuild/bazel/pull/19940#discussion_r1608124067 and https://github.com/bazelbuild/bazel/pull/19940#discussion_r1608069861

[PikachuHyA]

hi @comius , the third patch is ready. please review.

[PikachuHyA]

LGTM in general, except an a lot of renames to keep it consistent with PR[1/5].

Please replace cpp20_modules with cpp_modules. Please replace deps_scanning with cpp_module_deps_scanning - so that there isn't a confusion what the deps are. Or maybe is should be dep_module_scanning?

Complete the following renaming.

• CPP20_MODULES_INFO -> CPP_MODULES_INFO
• CPP20_MODULES_DDI -> CPP_MODULES_DDI
• CPP20_MODULES_MODMAP -> CPP_MODULES_MODMAP
• CPP20_MODULES_MODMAP_INPUT -> CPP_MODULES_MODMAP_INPUT
• CPP20_MODULE_OUTPUT_FILE -> CPP_MODULE_OUTPUT_FILE
• CPP20_DEPS_SCANNING -> CPP_MODULE_DEPS_SCANNING
• CPP20_MODMAP_FILE -> CPP_MODULE_MODMAP_FILE
• deps-scanner -> cpp-module-deps-scanner

How should I handle the singular and plural forms? Currently, CPP_MODULES_INFO, CPP_MODULES_DDI, CPP_MODULES_MODMAP, and CPP_MODULES_MODMAP_INPUT use the CPP_MODULES_XXX format, while others use the CPP_MODULE_XXX format. Should we unify them into the CPP_MODULE_XXX format?

Additionally, c++-module-compile and c++-module-codegen already exist. How should we handle c++20-module-compile and c++20-module-codegen?

[PikachuHyA]

@comius ping

[PikachuHyA]

@trybka ping

[peakschris]

@trybka we are keenly waiting the modules feature, is there anything that can be done to help expedite the review of this and other MRs? cc @comius

[trybka]

Apologies. I had some personal stuff take my attention this past week. I will prioritize this review (and the other splits if they are ready) this week.

I appreciate your patience as I am ramping up on this work.

[PikachuHyA]

I'm testing https://github.com/bazelbuild/bazel/pull/22553 and find a bug in this patch due to https://github.com/bazelbuild/bazel/pull/22743 merged

Changes

• squash old commits to 1 commit
• rebase to the latest master branch
• update compiler_input_flags_feature and compiler_output_flags_feature

[peakschris]

@comius @trybka can this go in now?

[PikachuHyA]

@trybka ping