Updated the STL branch

[TurkeyMan]

Resurrected @gchatelet's STL branch

[dlang-bot]

Thanks for your pull request and interest in making D better, @TurkeyMan! We are looking forward to reviewing it, and you should be hearing from a maintainer soon. Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the annotated coverage diff directly on GitHub with CodeCov's browser extension
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + druntime#2211"

[TurkeyMan]

Totally depends on: ~~https://github.com/dlang/dmd/pull/8341~~ ~~https://issues.dlang.org/show_bug.cgi?id=18954~~ Non utf-8 strings depends on: https://github.com/dlang/dmd/pull/8342

[gchatelet]

This is awesome that you get some time to work on this. Wow everything is mangled correctly now. So cool!

[TurkeyMan]

Yeah, I've been working on the mangling the C++ compatibility in DMD for weeks! This seems like a good test that we're reaching the goal.

PR's welcome! :P

[TurkeyMan]

There's crap loads of methods missing from those classes still. And basically every ptr+len method should get an additional extern(D) + T[] overload to be useful in D.

[gchatelet]

Yeah, I've been working on the mangling the C++ compatibility in DMD for weeks! This seems like a good test that we're reaching the goal.

I've been there. Did you manage to get all the tests here pass?

[TurkeyMan]

I didn't know they existed! You're hiding them in your fork ;)

[wilzbach]

BTW not sure whether you are aware of it, there's also an existing PR for std.pair: https://github.com/dlang/druntime/pull/1802

[TurkeyMan]

Yeah I think this is C++... lower_snake_case is what's expected. There's no indicator that a method is extern(C++) or extern(D), especially if you use intelli-sense or whatever. I expect more local-D-inlines to appear in these for perf over time.

[Geod24]

I think we should restrict ourselves to the strict requirement for compatibility for the moment. Until someone depends on it, there's no point in beautifying it.

[TurkeyMan]

'beautifying' what?

[Geod24]

Blocked by https://issues.dlang.org/show_bug.cgi?id=16479 (unless you want to merge this without tests ;) )

[atilaneves]

I don't think the different C++ standard library implementations vary much, but it might be an ongoing effort to actually get things to work (by which I mean "not have bugs" - I'm sure that vec.push_back will work). In my opinion the only way to do this properly is through dpp since then the headers are translated on the fly.

[TurkeyMan]

The implementations don't vary at all from the front-end perspective... that's the definition of 'standard' ;) The low-level functions and struct's are completely different though. Those need to live in version blocks, then get wrapped up into a form the front-end API can make use of.

I'm not sure druntime should depend on dpp, I think it's too important to have a controversial dependency like that. I think there's a path here where we just support the structs and extern to the low-level functions (allocation, and ABI related stuff) inside the version blocks, then the mostly-trivial front-end methods can just be implemented in D using the extern stuff.

[Geod24]

I'm not even sure this should live in Druntime TBH. It will bring its own set of maintenance issues, and I'm not sure tying those to the release schedule / requirements of druntime makes sense. We do have the C binding here, but they are used by the runtime. I think an (officially supported) dub package would be better for everyone.

[TurkeyMan]

Well... it's already there.

dub... sigh

[jacob-carlborg]

We do have the C binding here, but they are used by the runtime.

I think quite a lot of the C bindings are not used by the runtime. I know Mike seems to prefer to not have any public bindings in druntime at all, C or C++. If the runtime needs to use an external library it should keep bindings to those private.

A separate project should provide public bindings to C, C++, etc.

[wilzbach]

I think @andralex wanted this to be in druntime, but I agree that doing this as a separate project is better because breaking changes are handled via SemVer - not a two-year long deprecation process.

[jacob-carlborg]

Yeah, where does it end? Since D can interface with Objective-C, does that mean we add bindings to the whole Apple SDK as well (rhetorical question).

[jmdavis]

The C bindings that druntime has are the bindings for the OS, and IMHO, it makes a lot of sense for them to be there. Binding random third party libraries (be they C or otherwise) on the other hand, really doesn't make sense. Those should be in 3rd party packages.

As for C++'s standard library, I don't know. In principle it would be nice to have all of that tightly integrated enough that you can trivially use C++ from D, but I don't know if it makes the most sense for them to be in druntime once you start considering some of the practical issues - especially since we have not taken the approach of outright building a C++ compiler into the D compiler.

[TurkeyMan]

especially since we have not taken the approach of outright building a C++ compiler into the D compiler

What issue are you trying to address, and how does that help with calling into the C++ library? Most of the C++ library is containers, which I agree with @andralex, should probably be just as available as printf. It's not like it's a dependency on an optional lib that the user has to install.

[Geod24]

It's not like it's a dependency on an optional lib that the user has to install.

Except it is. And different system will have different implementations and/or version, which will make it all the harder to test and maintain. Over the short lifespan of extern(C++) we've already encountered some issues, for example the dual ABI (and in general, inline namespaces) are not handled correctly.

[atilaneves]

@TurkeyMan The front-end (i.e. the interface) might not change, but the layouts might. That's also important to declare properly. And that's ignoring that there's more than one C++ version, and the interface to the standard library most definitely does change between them. Which version of std::vector should we declare? The differences are mostly additive, but not always.

Then there's the fact that the headers look differently depending on preprocessor defines. Should we have version blocks for all of those despite the combinatorial explosion? And again, for which version (the headers change as compilers get updated)?

I'm not even getting close to suggesting that dpp be part of the runtime, I'm just saying that trying to put the C++ stdlib in druntime has many, many issues.

This isn't to say it shouldn't be done. But if it is, it will be lacking.

[TurkeyMan]

I appreciate every point you make. I never said it was simple, but I also think you might be slightly over-complicating it though.

All D compilers have an implied complementary C++ compiler; the D compilers extern(C++) ABI support assumes a particular complementary C++ compiler. We should only support 'current' C++ as defined by that complementary C++ compiler, and we should assume we link against STL for that same toolchain. (no point supporting moments in STL's ABI that extern(C++) doesn't support!)

For GDC, that's easy, since a version of GDC is married to a version of GCC. For any GDC build, core.stdcpp should work with whatever GCC doing.
For LDC, I'd probably track the defaults for Clang as built against at that same LLVM revision. I think that's likely the least surprising for users.

DMD is curious; on Windows, just support what's current, ie, what MS is currently supporting, since that's what DMD tracks. Posix DMD is the only question mark... what C++ compiler is DMD most likely to be used in conjunction with? (What C++ compilers are guaranteed ABI compatible with DMD? is it DMC?), and choose that compiler. This is the only case where I can see an awkward STL version situation potentially emerging, and hopefully the number of versions remains low.

Obviously the unit tests would prove core.stdcpp works against the specified complementary C++ compiler, and nothing more, and we clearly advertise the C++ that is considered complementary to the given D compiler.

So, since extern(C++) defines a large part of the ABI support, and is tested with the compilers unit-tests, the STL support that makes use of the compiler is intrinsically attached to the compiler version that it was tested against. They are interlocked, and so they need to release together.

If core.stdcpp was a separate library, it wouldn't correctly track developments in extern(C++), people would experience ABI mismatches in a somewhat stochastic way.

[atilaneves]

When the complimentary compilers are gcc and clang, passing -std= can possibly change the layout and interface of everything in the standard library. Only cl and dmc fix the C++ version (I'm not even 100% sure cl doesn't have an option anymore). The D declaration would have to match whatever standard version the programmer uses to compile the C++ instantiation.

For all I know the mangling changes and you'd get linker errors. I'm not sure, but I also wouldn't be surprised.

[atilaneves]

Oh, and keeping the D declarations of C++ standard library classes up-to-date with the corresponding compiler would be painful.

[TurkeyMan]

Right, and we may want to support -std in the future, but I'd start by officially supporting what's current.

There are some practical truth's that help us:

• GCC/Clang don't fuck with the STL data structures in different build configs like MSVC; the only ABI change I know of in the last decade that may affect the STL struct compatibility is RVO, but that consideration is a compiler feature, and not a problem for core.stdcpp. Do you have any reason to believe -std may affect STL struct compatibility beyond RVO?
  • Solving for this would require a compiler option like -extern_cpp_no_rvo
• MSVC do fuck with the STL data structures, but they are not affected by anything like -std; so I propose we track the current MSVC STL.
• If there is compelling reason to support an old MS-STL data struct layout via a version() in the future, we can worry about and debate the merit of that at that time.
  • MS are getting much better at ABI stability! Hopefully conventional wisdom from the past is out of date. 2015 and 2017 are fully ABI compatible; do you know of any live projects on older toolchains? MS are good at motivating developers to upgrade.

Oh, and keeping the D declarations of C++ standard library classes up-to-date with the corresponding compiler would be painful.

But necessary, fortunately it's very easy to unit-test!

[jacob-carlborg]

Clang recently (6.0.0) switched from GNU++98 to GNU++14. I’m not a C++ expert but this might have changed the ABI because I don’t think the small string optimization is allowed anymore.

https://github.com/llvm-mirror/clang/commit/466d8da5f89b1a780f735c86f414fa69ce63221b

[TurkeyMan]

That change will probably have also enabled RVO by default, which is awesome, because it will match D now ;)

[jacob-carlborg]

Another problem is that on macOS using Clang bundled with Xcode, it's difficult to know which upstream version Apple's fork corresponds to.

$ clang --version
Apple LLVM version 9.1.0 (clang-902.0.39.2)