Tracking Issue for constexpr Containers: P0784, P0980, P1004

[mnatsuhara]

Last updated 1/11/2022

📄 Relevant Papers

• WG21-P0784 (#37): Library Support for More constexpr Containers
• WG21-P0980 (#43): constexpr std::string
• WG21-P1004 (#45): constexpr std::vector

🚧 Status

The various papers involved in this broader feature are in various stages of completion:

• WG21-P0784 implemented by #1369 - done ✔️
• WG21-P0980 being implemented by #1502 - done ✔️
• WG21-P1004 being implemented by #1407 - done ✔️
• #1546 implementing some common machinery working toward #43, #45 - done ✔

🎈 Fixed Clang Bugs

Resolved in Clang 12

• LLVM-48587: is_constant_evaluated() can incorrectly evaluate to true in the destructor of a non-constexpr variable
  • Fixed and cherry-picked to Clang 12 🟢
• LLVM-48606: Clang rejects creation of struct with mutable member during constant evaluation - blocking for Clang ❌
  • Believed to be fixed in PR48606 and cherry-picked to Clang 12 here.
• LLVM-48582: Clang crashes compiling the MSVC standard library
  • Fixed and cherry-picked to Clang 12 🟢

Resolved in Clang 13

• LLVM-49342: -Wdangling-gsl incorrect warns on return by value of std::string::iterator() -= n - workaround applied ⚠
  • Fixed 🟢
  • Note that workaround is not transition; it will be kept even after bug fix for NRVO optimizations.

🐞 Active C1XX Bugs (MSVC Front-End)

• DevCom-1487178 - cl permits object reference to object outside of its lifetime during constant evaluation - not blocking 🟢
  • Microsoft VSO-1361413
• DevCom-1318307 - ICE during constexpr evaluation - not blocking for MSVC, found workaround ⚠️
  • Preprocessed repro added (Microsoft VSO-1270433)
• Microsoft VSO-1269037 - Compiler ICE on reference to pointer in constexpr - not blocking for MSVC, have workaround ⚠️
  • Workaround: next = &(*next)->s.p; --> U* temp = *next; next = &temp->s.p;
• DevCom-1347020 - MSVC accepts mutating of const variables during constant evaluation - not blocking 🟢
  • Microsoft VSO-1283648
• DevCom-1348647 - cl permits copying into memory before object lifetimes have begun - not blocking 🟢
  • Microsoft VSO-1284363
• Microsoft VSO-1284799 - constexpr ICE: binding a reference to a temporary emits Assertion failed: Nerrors > 0 - not blocking for MSVC, have workaround ⚠️
  • Workaround: _INLINE_VAR constexpr _Fake_allocator _Fake_alloc{};
• DevCom-1516290 "C++20 constexpr dynamic allocation doesn't properly understand std::construct_at()"
  • Affects vector<Base*>, no workarounds in STL product/test code.
  • Microsoft VSO-1388781
• DevCom-1540737 "constexpr std::string (with SSO) bug with user-defined char-like type"
  • Discovered in #1735
  • Microsoft VSO-1412925
• DevCom-1634430 "cl incorrectly rejects call to std::{ranges::}construct_at when passed a string literal as argument"
  • Discovered in #2467

🎉 Fixed C1XX Bugs (MSVC Front-End)

• DevCom-1266506 - constexpr destructors disallowed in literal types in parameters, return types
  • Fixed in 16.9 Preview 3 ✔️
• VSO-1256895 - constexpr does not evaluate explicit dtor call
  • Fixed in 16.9 Preview 3 ✔️
• VSO-1256891 - constexpr does not respect lvalue reference to class temporaries
  • Fixed in 16.9 Preview 3 ✔️
• DevCom-1318357 - ICE after adding constexpr to std::string
  • Microsoft VSO-1270432, fixed by Microsoft MSVC-PR-303657 in 16.10 Preview 1. ✔️
• DevCom-1318341 - ICE ~~in Back-End~~ using constexpr std::string
  • Microsoft VSO-1269894, fixed by Microsoft MSVC-PR-301160 in 16.10 Preview 1. ✔️
• Microsoft VSO-1275530 - [constexpr] Temporaries created for functions do not have a ctor called (and have a redundant dtor)
  • Fixed by Microsoft MSVC-PR-302562 in 16.10 Preview 1. ✔️
• DevCom-1333853 - 16.9p3 std::construct_at requires a copy constructor
  • Microsoft VSO-1276453, fixed by Microsoft MSVC-PR-303605 in 16.10 Preview 1. ✔️
• DevCom-1336816 - cl errors on constexpr array of std::string
  • Microsoft VSO-1277619, fixed by Microsoft MSVC-PR-303915 in 16.10 Preview 1. ✔️
• DevCom-1336819 - cl constexpr temporary passing issue
  • Microsoft VSO-1277640, fixed by Microsoft MSVC-PR-304321 in 16.10 Preview 1. ✔️
• Microsoft VSO-1272857 - [constexpr] Push state destroys formal on function entry
  • Fixed by Microsoft MSVC-PR-302273 in 16.10 Preview 1. ✔️
• DevCom-1328628 SFINAE bug when a default argument is protected
  • Reported with a fully reduced test case.
  • Encountered when mirroring #1546 to the MSVC-internal repo (the compiler uses std::optional with RapidJSON which contains this code pattern).
  • Microsoft VSO-1274031, fixed by MSVC-PR-311059 in 16.10 Preview 3 ✔️
• DevCom-1331017 - cl does not make special member functions implicitly constexpr - not blocking for MSVC, have workaround from @cdacamar ⚠️
  • Microsoft VSO-1275269, fixed by MSVC-PR-345557 in 17.1 Preview 1 ✔️
• DevCom-1633947 "cl erroneously detects an uninitialized read in an assignment during constant evaluation"
  • Discovered in #1735

🦗 Active EDG Bugs

• Microsoft VSO-1273296 - EDG: complaint about _wassert not constexpr - large portions of tests disabled for EDG 😢
  • Bugs that are different expression of same issue: Microsoft VSO-1273365, Microsoft VSO-1273386, Microsoft VSO-1274387
  • Remaining unfixed parts are same issue: Microsoft VSO-1273381 - EDG: error about a non-literal type in constexpr function
    • Fixed partially by MSVC-PR-302349.

🥳 Fixed EDG Bugs

• DevCom-1318318 - ICE during constant evaluation with /BE - related to std::destroy
  • Reported upstream (Microsoft VSO-1270011, EDGcpfe/23820)
  • Fixed by VS-PR-303727 in 16.10 Preview 1 ✔️
• DevCom-1318321 - cl Incorrectly rejects class with destructor for constant evaluation with /BE
  • Reported upstream (Microsoft VSO-1269976, EDGcpfe/23819)
  • Fixed by VS-PR-302349 in 16.10 Preview 1 ✔️
• Microsoft VSO-1285539 - EDG ICE with constexpr std::string in do_constexpr_std_construct_at
  • Reported upstream (EDGcpfe/23990)
  • Fixed by VS-PR-310478 in 16.10 Preview 2 ✔️
• Microsoft VSO-1601168 - EDG's __builtin_memcmp emits bogus errors with move-constructed strings and constexpr dynamic allocations
  • Reported upstream (EDGcpfe/25606)
  • Fixed by VS-PR-422901 in 17.4 Preview 3 ✔️

📑 Compiler Approaches to constexpr Dynamic Allocation

(from @cdacamar in #1532) Note that [expr.const]/5 states that an expression E is a core constant expression unless the evaluation of E would [...] would evaluate one of the following:

• a new-expression ([expr.new]), unless the the selected allocation function is a replaceable global allocation function ([new.delete.single], [new.delete.array]) and the allocated storage is deallocated within the evaluation of E;
• a delete-expression ([expr.delete]), unless it deallocates a region of storage allocated within the evaluation of E;
• a call to an instance of std::allocator::allocate ([allocator.members]), unless the allocated storage is deallocated within the evaluation of E;
• a call to an instance of std::allocator::deallocate ([allocator.members]), unless it deallocates a region of storage allocated within the evaluation of E;
• [...]

So, the expression ::operator new is not a new expression, it is a function call to a global operator new() which is not a constexpr function.

• Clang has chosen to allow ::operator new in a constexpr function if it is reached transitively from an evaluation of std::allocator::allocate.
• MSVC has chosen to do "fairy magic" (🧚) when it sees std::allocator::allocate, std::allocator::deallocate, but does not allow ::operator new.
• EDG (Intellisense) appears to allow ::operator new, but we choose to have the EDG logical paths mirror those for MSVC so they work in tandem.

Development notes, preserved for history

👣 Next Steps (as of Jan 2021)

• After getting the bug fixes in 16.9 Preview 3, @miscco discovered several new bugs in MSVC FE and BE and EDG (see Active C1XX Bugs, Active MSVC Back-End Bugs, and Active EDG Bugs).
• We are currently blocked for Clang on Bug 48606: Clang rejects creation of struct with mutable member during constant evaluation - this bug report has not yet been validated by a Clang developer, though @cdacamar has confirmed that this seems to be a Clang bug.
  • This bug is blocking in our iterator debug machinery, see further discussion here. In particular, we need to make some members of our debug iterators mutable, but Clang currently does not allow this due to this bug.
  • More background on the use case:
    • Because global vector variables can exist, we need to enable situations where a variable is constant initialized (so std::is_constant_evaluated() evaluates to true in the constructor), but not necessarily constant evaluated (so std::is_constant_evaluated() evaluates to false in subsequent calls to other member functions). You can see the discussion here.
    • This means that we cannot simply disable all iterator debugging machinery if std::is_constant_evaluated() is true in these containers in general. Hopefully, a fix for Bug 48606: Clang rejects creation of struct with mutable member during constant evaluation can be forthcoming soon, in which case we will be able to move forward with a Clang-facing implementation.
    • If a Clang dev determines that the reported bug is not valid and their behavior is Standards-conforming, we should (1) fix the Standard to allow mutable in this context, or (2) investigate whether requested an additional built-in to detect whether we are in a "constant initialization" context vs. "constant evaluation" context could also help us mitigate the debug iterators issue.
    • NOTE: More discussion and clarification here. Any global string variable will end up using a dynamic initializer as constant initializers will be off-limits. This is because we don't have non-transient allocations and all constant initializers for string now do constant-time allocation (since SSO is turned off if is_constant_evaluated()). So, the motivating argument for keeping the debug iterator proxy machinery active during constant evaluation does not apply to string, but the same iterator logic is used by many of the containers. Only disabling the proxy iterator things for string would require duplication of a lot of code. Given that there are plans to update our debug iterator machinery in vNext anyway, the effort required to disable debug iterators during constant-time for string does not seem worthwhile.

🖥️ Testing (as of Jan 2021)

• Since there have been a number of compiler issues discovered through the implementation of these features, we are going to enable testing the features' implementation in the internal test harness (via #ifdef MSVC_INTERNAL_TESTING) so as to discover and fix compiler issues more quickly.
  • As we are able to consume the compiler fixes in public previews, we will be able to enable full testing and remove the dependency on MSVC_INTERNAL_TESTING.
• Current status:
  • P1004R2_constexpr_vector tests are passing internally with the updated compiler -- #1690 enables the tests internally. When we consume the updated compiler with test fixes, we will be able to remove the defined(MSVC_INTERNAL_TESTING) entirely.
  • P1004R2_constexpr_vector_bool tests have all compiler bugs fixed internally, but there is a pre-existing issue in the debug iterator machinery that needs investigation before we can enable these tests.
  • P0980R1_constexpr_string tests have all compiler bugs fixed internally, and will already have defined(MSVC_INTERNAL_TESTING) throughout test code to enable these tests internally as soon as we merge.

[miscco]

Thanks for the amazing writeup!

[cdacamar]

MSVC has chosen to do "fairy magic" (🧚) when it sees std::allocator::allocate, std::allocator::deallocate, but does not allow ::operator new.

Fairy magic might be appropriate :). What the compiler actually does is elide calls to those functions (along with std::construct_at, std::destroy_at and friends) so that their body does not matter--since it will almost certainly do non-constexpr friendly things such as calls to ::operator new.

[AraHaan]

I mean why don't we use constexpr in the global new and delete operators by default too?

[Blackh0leZ]

Great bookkeeping for feature implementation quality, Miya!

[mnatsuhara]

I mean why don't we use constexpr in the global new and delete operators by default too?

This was addressed in the STL discord channel as well, but for the sake of completeness I will echo the reasoning here. Allowing constexpr new/delete is not technically permitted by the Standard. To quote @cdacamar, "One possible reason for not making the global new constexpr is because it does not establish any kind of object lifetime, it just allocates bytes. The abstract machine in C++ is all about working with objects which have lifetime and blobs of bytes do not really fit into that model very well."

Unless new and delete are made constexpr in the Standard, we will not be pursuing that approach.

[miscco]

🐞 Active C1XX Bugs (MSVC Front-End)

Added DevComm-1347020 - MSVC accepts mutating of const variables during constant evaluation

[AraHaan]

@mnatsuhara

• Bug 49342: -Wdangling-gsl incorrect warns on return by value of std::string::iterator() -= n
  • Accepted fix here: https://reviews.llvm.org/D97605

[miscco]

🐞 Active C1XX Bugs (MSVC Front-End) There is another compiler bug in the constexpr SSO pr (#1735). I have filed it here https://developercommunity.visualstudio.com/t/cl-erroneously-detects-an-uninitialized-read-in-an/1633947

[miscco]

🐞 Active C1XX Bugs (MSVC Front-End) And another one destilled from #2467 https://developercommunity.visualstudio.com/t/cl-incorrectly-rejects-call-to-stdrangesconstruct/1634430

[frederick-vs-ja]

VSO-1601168 ("EDG's __builtin_memcmp emits bogus errors with move-constructed strings and constexpr dynamic allocations") seemly should also be tracked here. It was discovered in #3057.

[CaseyCarter]

Someone with access to the Microsoft-internal bug database needs to make a pass over the OP and ensure everything is up-to-date. The "Active EDG bugs" all seem to be closed and resolved, for example. If we can remove any and all workarounds, we can close this issue.