[WIP] Proof of concept for versioning the standard library

[andralex]

This PR introduces a very limited v2 of the standard under the name std.v2. There are only very few artifacts in there. Essentially, this PR just showcases how mismatch can be versioned so as to accommodate autodecoding in v1 and no autodecoding in v2.

To that end, v2 supports the following artifacts: std.v2.range.primitives.empty, std.v2.range.primitives.front, std.v2.range.primitives.popFront, std.v2.range.primitives.isInputRange, std.v2.meta.allSatisfy. Some of these are simply pulled from v1, others are redefined with new meaning.

The characteristics of this approach:

• ZERO duplication of implementation.
• NO version/static if hell.
• NO patching or git cherry-picking across versions hell.
• Total control and flexibility with regard to inheriting symbols across versions with unchanged semantics, or redefining them with changed semantics.
• Interoperability across versions, including incremental migration.

The main challenges are:

• Documentation generation must be improved carefully: some documentation must be automatically generated multiple times across versions, some must be overridden, and probably some must be assembled with e.g. v1 + notes specific to v2.
• Language support for import must be improved. Right now the mixin imports are rather awkward.

[dlang-bot]

Thanks for your pull request, @andralex!

Bugzilla references

Auto-close	Bugzilla	Severity	Description
✗	22596	normal	The "publictests" target runs unittests at the top-level namespace so they don't have access to

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + phobos#8309"

[Imperatorn]

Whatever happens to this, I just want to say: a real attempt is worth 10x more than just discussing forever. It's 's trying to move things forward. Perfection is only found in the platonic universe, once brought out in the physical world, imperfections will always be noticeable.

Brace yourself, v2 is coming

[Imperatorn]

  | std/v2/range/primitives.d(15:24)[warn]: A public function needs to contain a Returns section.   | std/v2/range/primitives.d(15:24)[warn]: Public declaration 'front' has no documented example.   | std/v2/range/primitives.d(27:6)[warn]: Parameter a isn't documented in the Params section.   | std/v2/range/primitives.d(27:6)[warn]: Public declaration 'popFront' has no documented example.   | std/v2/algorithm/comparison.d(13:1)[warn]: A unittest should be annotated with at least @safe or @system   | std/algorithm/comparison.d(1851:10)[warn]: Public declaration 'canon' is undocumented.

[adamdruppe]

On Mon, Nov 01, 2021 at 06:41:53PM -0700, Andrei Alexandrescu wrote:

binaryFun with a string predicate imports a bunch of std stuff (including std.range). You need to migrate that as well.

Interesting. I'm not so sure. If std.functional.binaryFun version 1 works as needed to implement v2's mismatch then mismatch is free to just use it without migrating it at all. It's a trivial case of code reuse.

This means user predicates will exhibit behavior inconsistent with the rest of the library.

My view is binaryFun ought to simply be removed. It adds only complication coming significantly short of pulling its weight. v2 is a good opportunity to remove this kind of bloated complexity.

But if it is present, surely the functions you call from inside it ought to behave similarly to everything else in std.v2?

[Imperatorn]

stdv2 should be on the same level as std so the directory structure will be the same

[andralex]

Another bug in the compiler revealed by unittests:

../dmd/generated/linux/release/64/dmd  -conf= -I../druntime/import  -w -de -preview=dip1000 -preview=dtorfields -preview=fieldwise -m64 -fPIC -O -release -defaultlib= -debuglib= -L-lpthread -L-lm -betterC -unittest -run generated/linux/release/64/betterctests/std_algorithm_comparison.d

issues the nonsensical error:

std/array.d(3312): Error: `TypeInfo` cannot be used with -betterC

The line is bogus and the error is bogus.

[Imperatorn]

Another bug in the compiler revealed by unittests:

../dmd/generated/linux/release/64/dmd  -conf= -I../druntime/import  -w -de -preview=dip1000 -preview=dtorfields -preview=fieldwise -m64 -fPIC -O -release -defaultlib= -debuglib= -L-lpthread -L-lm -betterC -unittest -run generated/linux/release/64/betterctests/std_algorithm_comparison.d

issues the nonsensical error:

std/array.d(3312): Error: `TypeInfo` cannot be used with -betterC

The line is bogus and the error is bogus.

I think we found out why this happens

Another bug in the compiler revealed by unittests:

../dmd/generated/linux/release/64/dmd  -conf= -I../druntime/import  -w -de -preview=dip1000 -preview=dtorfields -preview=fieldwise -m64 -fPIC -O -release -defaultlib= -debuglib= -L-lpthread -L-lm -betterC -unittest -run generated/linux/release/64/betterctests/std_algorithm_comparison.d

issues the nonsensical error:

std/array.d(3312): Error: `TypeInfo` cannot be used with -betterC

The line is bogus and the error is bogus.

For reference

https://issues.dlang.org/show_bug.cgi?id=20715

[andralex]

@Imperatorn @dukc any ideas for workarounds for https://issues.dlang.org/show_bug.cgi?id=20715?

[Imperatorn]

@Imperatorn @dukc any ideas for workarounds for https://issues.dlang.org/show_bug.cgi?id=20715?

I can try to take a look at this, I just have a cold atm so my head is in a suboptimal configuration

[dukc]

Other than avoiding using destructors or disabling copy constructors in -betterC, not really. @Imperatorn Did you find out what -betterC test precisely triggers the typeinfo bug?

[andralex]

Also submitted https://issues.dlang.org/show_bug.cgi?id=22579

[12345swordy]

* Language support for `import` must be improved. Right now the `mixin` imports are rather awkward.

Can you give us an example of what improvements you have in mind?

I have something like this in mind: import std version 2.x

[andralex]

@12345swordy didn't think of it much yet. Your idea looks a bit cute for me - what's the convention on the version formatting etc. I'd rather have something string-based.

[andralex]

Related: https://issues.dlang.org/show_bug.cgi?id=22596

[MoonlightSentinel]

FYI the current implementation breaks the navigation in the documentation, see e.g. std.algorithm and std.algorithm.searching

[andralex]

FYI the current implementation breaks the navigation in the documentation, see e.g. std.algorithm and std.algorithm.searching

Affirmative. It's a bug I'll submit in a few hours that needs to be fixed before this PR can be merged.

[MoonlightSentinel]

Good. But it's a red flag that an implementation detail like canon leaks into the documentation at all.

[dukc]

I think you should stop adding new functions and get what you have already made merged first. That way others can also begin porting the functions.

[andralex]

I think you should stop adding new functions and get what you have already made merged first. That way others can also begin porting the functions.

I wish things worked out that way, and that was my hope starting this. However, there are a couple of problems that make that impossible. Loosely in decreasing order of importance:

1. The restructuring of the code makes the generated documentation unusable.
2. In order to figure out what documentation generation features to ask for, we need to figure what code structure scales best
3. In order to figure out scaling, we need to experiment at some scale so as to assess the issues when multiple interdependent modules are being versioned.
4. In the process of doing all that, there are bugs in the compiler that need fixing before merging.

So... not a simple or quick process.

[dukc]

1. The restructuring of the code makes the generated documentation unusable.

2. In order to figure out what documentation generation features to ask for, we need to figure what code structure scales best

3. In order to figure out scaling, we need to experiment at some scale so as to assess the issues when multiple interdependent modules are being versioned.

4. In the process of doing all that, there are bugs in the compiler that need fixing before merging.

Just disable the docs for now. It's going to require future PRs anyway before v2 is ready for any serious use, and we're definitely not going to worry about breaking the code of any experimenters. So no need to do it all in one monolith PR.

[andralex]

Just disable the docs for now.

How do you mean that?

[dukc]

Anyone who knows a bit about docs, what would be the best way to hide the documentation of V2 for now without affecting V1?