Support Parts/augments with an "external" (generated) path

[rrousselGit]

Problem: It is tedious to move generated code to a different place than next to the source

One of the most common complaints with code-generation is its pollution of the source code. It pollutes both the file explorer and code reviews.

One of the solution to those is to move generated code in a separate ignored folder (such as .dart_tool). But there's one massive inconvenience with doing this today: parts need to point to the generated file by hand.

Given:

pubspec.yaml
.dart_tool/
generated/libsrc/folder/example.g.dart
lib/src/folder/example.dart

We end with:

// lib/src/folder/example.dart

part '../../../../generated/lib/src/folder/example.g.dart';

Problem: Renaming a file doesn't rename its parts

A simpler one (that's possibly solvable today using tooling changes) is, given file.dart which defines part 'file.g.dart, then renaming file.dart -> new_name.dart will not rename part 'file.g.dart' -> part 'new_name.g.dart'

Proposal: Support external parts ; parts that point to an equivalent file inside .dart_tool

The issues mentioned above are ultimately caused by needing a path relationship between the source and its associated generated code.

Could we introduce some form of convention for when a part is meant to be generated by tooling? This would be a path-less part.

We could imagine:

// lib/src/folder/example.dart

external part 'name.dart';

And the compiler would look for .dart_tool/lib/src/folder/name.dart, where the path to name.dart is obtained by concatenating lib/src/folder/example.dart, .. and the part name (name.dart)

[lrhn]

Another problem with moving files into the .dart_tool directory is that they are outside of the lib/ directory. That means that files inside the lib/ directory cannot refer to them, not while having a package: URI, because package: URIs can only refer to files inside the lib/ directory. Relative paths cannot exit the current lib/ directoy, absolute paths are not portable, and there is a good chance you'll eventually just get a compile-time error if you try to refer out of lib/.

The

// lib/src/folder/example.dart, aka package:thispkg/folder/example.dart
part '../../../../generated/lib/src/folder/example.g.dart';

will resolve to package:thispkg/generated/lib/src/folder/example.g..dart when the example.dart file is imported using the package: URI (which you always should for files in lib/).

Even with an external part, there would have to be some sort of URI for the part file, otherwise it cannot have relative imports itself (see "parts with imports"!), and its part of needs to point somewhere too, but that can use an absolute package: URI.

The external part 'name.dart'; introduces a new namespace, for "external parts", similar to the package: namespace for package references. Might as well be direct and call it generated:thispkg/example.g.dart, and then the compiler will have to resolve that somehow. (Heck, maybe even trigger a code generator if there is no generated file of that name.) Then the generated file would still be considered part of the thispkg package (which is important for things like default language version), and it would be considered part of the files available to packages depending on the thispkg package. Which means that the generated files, or instructions on how to generate them, must be uploaded to Pub.

The way Dart, the language, is defined, you don't need anything outside of lib/ of the packages you depend on (normal depends, not dev-depends), so technically it doesn't even need to exist on your computer when you compile. (Pub doesn't have to download the entire package, it can just download the lib/ directory and the pubspec.yaml. Everything else isn't needed for a dependency.)

If a client of a package has to generate code into the package before they can use the package, then the dependencies needed for generating code is no longer dev-dependencies, they're just (compile-time) dependencies. The client package inherits those dependencies, and can get conflicts in version solving them. That's why it's better to check in the generated code, not the generator.

I think it's more viable to just have a .gen/ directory inside lib/ or src/lib/ for the generated files, and make tools recognize such a directory as special, or recognizing a file name of *.g.dart as generated.

[rrousselGit]

I don't mind if it's a lib/.gen directory. It doesn't matter that much in the grand scheme of things

The point of not wanting to have a path relationship between the file and its generated code still stands :)

Even with an external part, there would have to be some sort of URI for the part file, otherwise it cannot have relative imports itself (see "parts with imports"!), and its part of needs to point somewhere too, but that can use an absolute package: URI.

I think it's not a case of "the part doesn't have an URI", but more "The URI should be synthetic". Whether it is implied by the compiler or hand typed is the same.

I'd argue that it'd be useful if we could straight up do:

external part; // without a name at all

Users of code generation don't really care about the name in the end.

[munificent]

Here's a probably terrible idea: We could loosen the restriction that requires part and part of directives to be symmetric. A (likely generated) file could declare that it is a part of some other library without that library having to point to it.

Of course the main downside is that you have something modifying your program with absolutely no direct linkage from the thing being modified to the thing that modifies it:

• Users reading the main file have no way of knowing that there is a generated file out there somewhere that is changing what they see.
• Tools like analyzer that need to traverse and find all files for a program have no direct way of knowing to look for that generated file. Instead, we'd have to assume something like all files in lib/ must be potentially part of the program.
• Tools like code generators that can't easily find all files in a directory and need to request individual specific files would have no ability to detect and pull in these files.

[rrousselGit]

Yeah that's why IMO having a convention here makes sense

Rather than "any file can augment any other file", I find the idea of lib/.gen/dir/a.dart can augment lib/dir/a.dart (and only this file)

This way the "assymetry" you described is predictable.

• Users reading the main file have no way of knowing that there is a generated file out there somewhere that is changing what they see.

If we want to completely remove the part, the IDE could show some synthetic code similar to the Widget guides.

The top of the file could say "there's a generated file. Click here to see it"

Kind of like macros worked

[lrhn]

A (likely generated) file could declare that it is a part of some other library without that library having to point to it.

How does the compiler know what it has the entire program?

We have (effectively) the restriction that a part must be in the same package as its containing file, and that if the library is in lib/, so must the part be. That limits the search to the package root directory of that package, and for package: URI-files, to the lib/ directory (or where it's specified in the package_config.json).

That doesn't work for files without a package. We'd have to say that this doesn't work for non-package files (like scripts). Probably not a big loss.

So, to compile a single file program like foo/tool/generate_table.dart, the compiler must search the entire foo package directory, transitively, and look in every file to see if it's Dart file that starts with part of '..../generate_table.dart';. Dart files don't have to end in .dart, so I do mean every file, unless we restrict this feature to files that end in .dart. (And then we could require them to end in .part.dart, to make discovery even easier.)

The file should specify that it's an implicit part file, say external part of '...' instead. Otherwise it will make explicit conditional parts impossible (or more complicated)

part "fool_default.part"
   if (dart.library.io) "foo_io.dart"
   if (dart.library.js_interop) "foo_js.dart";

This syntax is allowed by enahnced parts, and I think it is one of the features that will be very useful.

If the compiler would blindly see that foo_default.dart, foo_io.dart and foo_js.dart are all part of 'foo.dart'; and include them all, things wouldn't work. Either mark the external parts as such, so that it's clear that these are not to be implicitly imported, or the compiler will have to check all the not taken conditions, and exclude all the files potentially referenced by those.

I think I'd rather just have code generation insert the part directive in original source code. Probably not something build-runner would like, though. I don't think it's big on editing the root source files, could cause a lot of churn if it cannot ignore the inserted parts.

[rrousselGit]

That's why I'd rather use conventions, such that the location of the generated file is at a predictable location in a hidden folder

[rmtmckenzie]

My biggest issue with the current code generation solution is that you need to manually write specific boilerplate to get it to work in every single file you use it in. Sure, it's not that much - but if you maintain a large project you've probably spent a not-insignificant of time writing the part file directives. Getting rid of that would remove a huge barrier to the adoption of code generation.

If code generation were treated as a more first-party part of the dart ecosystem such that these part directives weren't needed for generated code, I think this could be made a lot smoother without introducing the same complexity that macros ran into. It could also be treated separately from normal part files - I imagine that would be more work to implement but would reduce overlap with things like conditional parts.

Building off of what Remi described & suggested (i.e. a matching path for each file with any code generation annotations), and having the complier include that by default, it could be possible to improve the developer experience significantly.

i.e. lib/code.dart could automatically include .dart_tool/gen/lib/code.dart#g and .dart_tool/gen/lib/code.dart#freezed. And if you really didn't want one included, there could be a way to ignore generated code although I think this could potentially be mitigated by using either composition or inheritance.

This would be a bit of a departure for dart tooling... but after all, wasn't that what the macros project about - making a much better developer experience at the cost of more tooling?