Support C++20 modules

[abergmeier]

Description of the problem / feature request / question:

Starting to look into C++ Modules. While it will earliest officially arrive in 2020, would be good to have support for that soonish. Heard there is some Module support inside of Google. Can/Will that be opensourced?

[hlopko]

Hi Andreas, everything regarding modules (only for clang) is open sourced already. The only missing piece is a CROSSTOOL that makes use of them and provides all necessary features. But I think it would be quite a challenge to write such CROSSTOOL without understanding the implementation. If I don't find a good movie on my flight to bazel conference I might take a look and hack something together. No promises though, since this would be a free time exercise, not something that is a priority for bazel right now.

[abergmeier]

Interesting. @mhlopko Can you perhaps elaborate a bit, what parts of CROSSTOOL would interact with modules? Personally I thought it would be more in the realm of cc-providers and additional actions for module validity checks.

[hlopko]

Yeah you are right, the implementation does deal with providers and actions, but all that logic is already in bazel and ready to be used. What is missing is a bunch of features (and maybe action_configs, I'm not sure) in the CROSSTOOL to pass the right flags for the right actions. Names of the features are hardcoded in bazel, so they need to match.

[hlopko]

So in other words, it's all there, just not very well documented.

[abergmeier]

just not very well documented

Like all else about CC toolchains you mean ;)

Thanks. I am trying to get a custom hermetic Clang toolchain running anyways. Perhaps will then try this out a bit.

[hlopko]

Exactly :) I hope I'll see you next week on the crosstool breakout session during bazel conference? :) Btw, just wrote this yesterday, I really hope I'm not one day late: https://github.com/bazelbuild/bazel/wiki/Yet-Another-CROSSTOOL-Writing-Tutorial

Anyway, don't hesitate to ask if you get stuck on the way. I'm also grateful for any input/comments/suggestions.

[abergmeier]

Since I do have problems with the include paths for libc (https://stackoverflow.com/questions/47078520/clang-toolchain-fails-for-system-libc-files), how does libc_top (inputsForLibc) work?

Especially since I cannot seem to be able to glob system files.

[ArekSredzki]

Now that C++20 Modules are nearing standardization, could we get some updates on the status of properly supporting them within Bazel?

I was fortunate to attend CppCon this year and was excited to see that many build systems were making progress on supporting modules. Unfortunately, myself and other Bazel users that I spoke to were concerned that there was no Bazel-specific information on the topic.

[hlopko]

We don't plan to work on C++ modules in 2019. We are discussing working on them in 2020 mostly in the context of objc and swift rules. We'll update this issue once we have more precise plans.

[ArekSredzki]

Thanks @hlopko, I'm sure that much of the C++ Bazel community will very much value support for them. Looking forward to hearing more.

[ctrysbita]

Any update here? :D

[ArekSredzki]

@hlopko Is there anyone that we could ping regarding the timeline for supporting C++20 modules? There appears to be significant community interest in this feature, it'd be good to set expectations :)

[hlopko]

@oquenchil and @c-mita are authorities there.

[ArekSredzki]

@oquenchil @c-mita Bump on the above? :)

[oquenchil]

It might be that this gathers steam at the beginning of next year but I can't promise anything. In any case, it would be clang modules, not C++20 modules.

Whoever is interested in clang modules, definitely thumbs up the original issue above and that might help with prioritization.

[technoir42]

it would be clang modules, not C++20 modules.

Should we create a separate issue for C++20 modules then? I'm only interested in those as I'm using MSVC on Windows.

[oquenchil]

Yes, you can create a separate issue for tracking C++20 module support. Be aware though that as it looks right now it will be a long time before we work on it, probably not next year.

[thii]

Can you rename this issue to "Support Clang modules"?

[rnburn]

In any case, it would be clang modules, not C++20 modules.

@oquenchil - By "Clang modules", do you mean Clang's non-standard module system that pre-dates C++20 modules?

Why target that instead of C++20 modules? Clang supports a lot of the standard module spec already (blog post) -- I expect enough that people could start using it.

[oquenchil]

A lot of the implementation for Clang modules already exists for the Google internal version of Bazel, the effort required to implement Clang modules in Bazel would be less than for C++20 modules. We are not discarding ever implementing the latter though, just that it would come at a later time.

[rnburn]

But why hijack this issue into a clang module feature request? The original poster clearly worded it for C++20 modules and I expect that's what most people following it want.

If someone's looking for clang's homegrown module system, can we let them submit a different issue for it? Is it even broadly used outside of Google?

C++20 module support may not be on Google's short-term roadmap, but there are a lot of project that would benefit from being able to use standard modules, so perhaps someone else might sponsor the work.

[ArekSredzki]

I second that C++20 module support would be far more beneficial to the community at this point. Many organizations use multiple toolchains, in which case, supporting Clang's module implementation brings little value.

I suspect that most people following this ticket are interested in C++20 modules since that's what it was created for. Could a separate ticket be created for Clang modules instead of renaming this one as that's a separate request with a niche application? 🙏

[ArekSredzki]

Thanks @oquenchil !

[det]

Hello! I took a shot at implementing Clang Modules in a local bazel repo as this ticket said that everything for modules is already open-sourced. I believe I have implemented most of the crosstool features nearly correct (but maybe not?) and I am running into a couple issues:

1. I'm not sure how module maps for the C++ stdlib and sysroot (libc) should be handled:
   • You must provide a crosstool.cppmap as the module_map for your cc_toolchain. It's not clear if this should include maps for both your c++ stdlib and sysroot.
   • Bazel doesn't ever seem to compile this module map into a pcm file. How are we supposed to acheive this?
     • Are you supposed to precompile a .pcm file and include it in your compiler files and add -fmodule-file=path/to/crosstool.pcm to the use_header_modules feature?
       • Can this even work since pcm files are sensitive to compile flags which can be changed on the command line?
2. Bazel seems to support 2 modes of modules support, with and without modules codegen. When using --features=header_module_codegen it will correctly try to compile the .pcm file to a .o file but it is not making available all the .pcm dependencies of the .pcm file that is being compiled. For example, if foo.pcm imports bar.pcm but when foo.pcm is being compiled to foo.pcm.o it will try to open bar.pcm which is not available in the sandbox and so will error. I also verified with aquery that the dependent pcm files are not provided as inputs to that compile action.

Except for the layering check, there is no documentation or examples to be found anywhere on the internet so I would appreciate any pointers as well as clarification of whether clang modules should currently be usable in the open source bazel. @hlopko @oquenchil

I am using bazel 3.7.1 and Clang 11.0.0 in my toolchain if that matters.

Thanks!

[EvanHonnold]

@hlopko @oquenchil Might you have a chance to take a look at @det 's comment above? I am running into many of the same issues and would be curious to know if there are any workarounds.

[hlopko]

I haven't yet, @c-mita, maybe you have worked with Bazel modules recently?

[oquenchil]

Hello! I took a shot at implementing Clang Modules in a local bazel repo as this ticket said that everything for modules is already open-sourced. I believe I have implemented most of the crosstool features nearly correct (but maybe not?) and I am running into a couple issues:

1. I'm not sure how module maps for the C++ stdlib and sysroot (libc) should be handled:

   • You must provide a crosstool.cppmap as the module_map for your cc_toolchain. It's not clear if this should include maps for both your c++ stdlib and sysroot.

   • Bazel doesn't ever seem to compile this module map into a pcm file. How are we supposed to acheive this?

     • Are you supposed to precompile a .pcm file and include it in your compiler files and add -fmodule-file=path/to/crosstool.pcm to the use_header_modules feature?

       • Can this even work since pcm files are sensitive to compile flags which can be changed on the command line?

2. Bazel seems to support 2 modes of modules support, with and without modules codegen. When using --features=header_module_codegen it will correctly try to compile the .pcm file to a .o file but it is not making available all the .pcm dependencies of the .pcm file that is being compiled. For example, if foo.pcm imports bar.pcm but when foo.pcm is being compiled to foo.pcm.o it will try to open bar.pcm which is not available in the sandbox and so will error. I also verified with aquery that the dependent pcm files are not provided as inputs to that compile action.

Except for the layering check, there is no documentation or examples to be found anywhere on the internet so I would appreciate any pointers as well as clarification of whether clang modules should currently be usable in the open source bazel. @hlopko @oquenchil

I am using bazel 3.7.1 and Clang 11.0.0 in my toolchain if that matters.

Thanks!

So I tried this out and I get a toy example to work with modules. I added the features to unix_cc_toolchain_config.bzl and made sure I was using clang with CC=clang. The module maps for system libraries should already be taken care of thanks to this.

You said that you are using 3.7.1, so I suspect the problem is that you didn't re-build bazel after making changes to unix_cc_toolchain_config.bzl. You have to re-build and use the custom binary. Please try that and let me know if you bump into any issues.

[rnburn]

I put together a project with bazel rules for working with C++20 modules:

https://github.com/rnburn/rules_cc_module

Here are some examples of how to use it

https://github.com/rnburn/rules_cc_module/tree/main/example

It requires a recent version of gcc and there are still some pieces to fill in; but I think with a bit more work, it could be made to handle most use cases. Let me know if anyone is interested in helping to finish the project.

[oquenchil]

That's interesting. Thanks for doing that Ryan.

Would you be interested in investigating how to add support for C++20 modules to the existing native rules? It would probably require a design doc and discussion.

[rnburn]

I actually think a first iteration for module support would be pretty easy.

1. Add a new rule cc_module that's defined something like this

cc_module(
      name = "module_a",
      module_name,  # specifies the module name, defaults to <name>
      src = "module_a.cc", # required, specifies the source file with "export module <module_name>"
      impl_srcs = [
           "a_impl1.cc",    
           ...   # optional list of implementation sources, they would specify "module <module_name>" with no export keyword
      ],
     deps = [
           # list of either modules or cc_libraries
     ],
)

cc_module would produce both a "compiled module interface" (e.g. module_a.cmi), something built from only src; and a standard archive (e.g. libmodule_a.a), formed from the object files for src and the impl_src's.

2. The other cc_* attributes would need to be made module-aware: For any cc_modules that they depend on, the compiler would need to be passed a mapping telling it where the CMIs are

<module1> <module_cmi1>
<module2> <module_cmi2>
... 

For gcc this can be done with the -fmodule-mapper=<map_file> argument; and for clang, this can be done with -fmodule-file=<module_name>=<module_cmi> arguments.

There are some other features you'd want to add at some point, like

• Support for producing header unit modules
• Support for module partitions (could be done by adding partition argument to cc_module)
• Maybe support for bundling multiple modules in a single rule
• Figure out what to do if you try to mix cc_modules compiled with options that would make them incompatible (e.g. -fno-exceptions)

But I don't think you need to worry about those for a first cut.