Code generation (metaprogramming) proposal v2

[Levi-Lesches]

trafficstars

This is my second draft of a proposal for code generation ("code gen" from now on) through specialized classes called "macros". The issue where this is being discussed is #1482 and my first proposal is #1507.

The key idea for me in code generation is that generated code should function exactly as though it were written by hand. This means that code gen should be strictly an implementation detail -- invisible to others who import your code. This further strengthens the need for code gen to be simple and expressive, so that any code that could be written by hand can be replaced by code gen, and vice-versa.

Another critical point is that code generation should not modify user-written code in any way. For this reason, part and part of are used to separate the human-written code from the generated. Partial classes (#252) help facilitate this, and this proposal heavily relies on them.

Definition of a macro

A macro is a class that's responsible for generating code. You can make a macro by extending the built-in Macro superclass. There are two types of macros: those that augment classes, and those that generate top-level code from functions. The Macro class consists of the generate function and some helpful reflection getters. The base definition of a macro is as follows:

// to be defined in Dart. Say, "dart:code_gen";

/// A class to generate code during compilation.
abstract class Macro {
  /// Macros are applied as annotations, see below.
  const Macro();

  /// The name of the entity targeted by this macro.
  String get sourceName => "";

  /// Generates new code based on existing code. 
  String generate();
}

Here are class-based and function-based macros, with Variable representing basic reflection.

Variable

/// A example of how reflection will be used. 
/// 
/// This class applies to both parameters and fields.
class Variable<T> {
  /// The name of the variable
  final String name;

  /// Any annotation applied to the variable.
  final Object? annotation;

  /// Represents a variable.
  const Variable(this.name, [this.annotation]);

  /// The type of the variable.
  Type get type => T;
}

ClassMacro

/// A macro that is applied to a class. 
/// 
/// Generated code goes in a partial class with the same name as the source.
abstract class ClassMacro extends Macro {
  const ClassMacro();

  /// Fields with their respective types.
  List<Variable> get fields => [];

  /// The names of the fields 
  List<String> get fieldNames => [
    for (final Variable field in fields)
      field.name
  ];

  // More helpers as relevant...
}

FunctionMacro

/// A macro that is applied to a function.
/// 
/// Generates top-level code since functions can't be augmented. If you want 
/// to wrap a function, do it using regular Dart.
abstract class FunctionMacro {
  const FunctionMacro();

  /// The body of the function. 
  /// 
  /// => can be desugared into a "return" statement
  String get functionBody => "";

  /// The positional parameters of the function.
  List<Variable> get positionalParameters => [];

  /// The named parameters of the function.
  List<Variable> get namedParameters => [];

  /// All parameters -- positional, optional, and named.
  List<Variable> get allParameters => [];

  // More helpers as relevant...
}

That being said, I'm not an expert on reflection so this is not really the focus of this proposal. I do know enough to know that dart:mirrors cannot be used, since it is exclusively for runtime. We want something more like the analyzer package, which can statically analyze code without running it. Again, the point is that code gen should work exactly like a human would, and the analyzer can be thought of as a second pair of eyes (as opposed to dart:mirrors which is a different idea entirely). Depending on how reflection is implemented, we may need to restrict subclasses of Macro, since that's where all the initialization happens.

Using macros

To create a macro, simply extend ClassMacro or FunctionMacro and override the generate function. All macros will output code into a .g.dart file. The code generated by a ClassMacro will go in a partial class, whereas the code generated by a FunctionMacro will be top-level. Because the function outputs a String, List<String>.join() can be a clean way to generate many lines of code while keeping each line separate. dart format will be run on the generated code, so macros don't need to worry about indentation/general cleanliness. To apply a macro, use it like an annotation on a given class/function. Here is a minimal example:

// greeter.dart
import "dart:code_gen";  // imports the macro interface

class Greeter extends ClassMacro {
  const Greeter();  // a const constructor for use with annotations.

  /// Generates code to introduce itself. 
  /// 
  /// This function can use reflection to include relevant class fields. 
  @override
  String generate() => 
    'String greet() => "This is a $sourceName class";';
}

// person.dart 

// Dart should auto-generate this line if not already present 
part "person.g.dart";

/// A simple dataclass. 
/// 
/// This is the source class for the generator. So, [Macro.sourceName] will be 
/// "Person". The class is marked as `partial` to signal to Dart that the 
/// generated code should directly modify this class, as though they were 
/// declared together. See https://github.com/dart-lang/language/issues/252
@Greeter()  // applies the Greeter macro
partial class Person { }

void main() {
  print(Person().greet());
}

// person.g.dart
part of "person.dart";

partial class Person {
  String greet() => "This is a Person class";
}

They key points are that writing the generate() method felt a lot like writing the actual code (no code-building API) and the generated code can be used as though it were written by hand. Also, the generated code is kept completely separate from the user-written Person class.

Commonly requested examples:

The best way to analyze any new feature proposal is to see how it can impact existing code. Here are a few commonly-mentioned use-cases for code generation that many have a strong opinion on.

JSON serialization

// to_json.dart
import "dart:code_gen";

/// Generates a [fromJson] constructor and a [json] getter.
class JsonHelper extends ClassMacro {
  const JsonHelper();

  /// Adds a Foo.fromJson constructor. 
  String fromJson() => [
    "$sourceName.fromJson(Map<String, dynamic> json) : ",
    [   
      for (final String field in fieldNames)
        '$field = json ["$field"]',
    ].join(",\n"),
    ";"  // marks the constructor as having no body
  ].join("\n");

  /// Adds a Map<String, dynamic> getter
  String toJson() => [
    "Map<String, dynamic> get json => {",
    for (final Variable field in fields)
      '"${field.name}": ${field.name},',
    "};",
  ].join("\n");

  @override
  String generate() => [fromJson(), toJson()].join("\n"); 
}

// person.dart
import "to_json.dart";

part "person.g.dart";  // <-- added automatically the first time code-gen is run

// Input class: 
@JsonHelper()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

// Test: 
void main() {
  final Person person = Person(name: "Alice", age: 42);
  print(person.json);  // {name: Alice, age: 42}
}

// person.g.dart
part of "person.dart";

partial class Person {
  Person.fromJson(Map<String, dynamic> json) : 
    name = json ["name"],
    age = json ["age"];

  Map<String, dynamic> get json => {
    "name": name,
    "age": age,
  };
}

Dataclass methods

import "dart:code_gen";

/// Generates [==], [hashCode], [toString()], and [copyWith].
class Dataclass extends ClassMacro {
  const Dataclass();

  /// The famous copyWith method
  String copyWith() => [
    "$sourceName copyWith({",
    for (final Variable field in fields)
      "${field.type}? ${field.name},",
    "}) => $sourceName(",
    for (final String field in fieldNames)
      "$field: $field ?? this.$field,",
    ");"
  ].join("\n");

  /// Overrides the == operator to check if each field is equal
  String equals() => [
    "@override",
    "bool operator ==(Object other) => other is $sourceName",
      for (final String field in fieldNames)
        "&& $field == other.$field",
    ";"
  ].join("\n");

  /// Implements a hash code based on [toString()].
  /// 
  /// You can use more complex logic, but this is my simple version. It also
  /// shows that standard functions can be generated with macros. 
  /// 
  /// Make sure this is interpreted as [Macro.hash], not [Object.hashCode].
  String hash() => "@override\n"
    "int get hashCode => toString().hashCode;";

  /// Implements [toString()] by printing each field and the class name. 
  /// 
  /// Don't name it toString()
  String string() => [
    "@override",
    'String toString() => "$sourceName("',
    for (final String field in fieldNames)
      '"$field = \$$field, "',
    '")";'
  ].join("\n");

  @override
  String generate() => [
    equals(),
    hash(),
    string(),
    copyWith(),
  ].join("\n");
}

// person.dart
import "dataclass.dart";

part "person.g.dart";

@Dataclass()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

void main() {
  Person alice = Person(name: "Alice", age: 42);
  print(alice.hashCode);
  print(alice.toString());
  final Person alice2 = alice.copyWith();
  final Person bob = alice.copyWith(name: "Bob");
  if (alice == alice2 && alice != bob) {
    print("Equals operator works");
  }
}

// person.g.dart
part of "person.dart";

partial class Person {
  @override
  bool operator ==(Object other) => other is Person 
    && name == other.name 
    && age == other.age;
  
  @override
  int get hashCode => toString().hashCode;
  
  @override
  String toString() => "Person("
    "name = $name, "
    "age = $age, "
  ")";

  Person copyWith({
    String? name,
    int? age
  }) => Person(
    name: name ?? this.name,
    age: age ?? this.age,
  );
}

Auto-dispose

// disposer.dart
import "dart:code_gen.dart";

/// An annotation to mark that a class should be disposed. 
class ShouldDispose { const ShouldDispose(); }
const shouldDispose = ShouldDispose();

/// Calls .dispose on all fields with the [ShouldDispose] annotation.
class Disposer extends ClassMacro {
  const Disposer();

  @override
  String generate() => [
    "@override",
    "void dispose() {",
      for (final Variable field in fields)
        if (field.annotation is ShouldDispose)
          "${field.name}.dispose();",
      "super.dispose();",
    "}",
  ].join("\n");
}

// widget.dart
import "disposer.dart";

part "widget.g.dart";  // <-- injected by code gen 

class MyWidget extends StatefulWidget {
  @override
  MyState createState() => MyState();
}

@Disposer()
partial class MyState extends State<MyWidget> {
  @shouldDispose
  TextEditingController controller = TextEditingController();

  // Not included in the dispose function
  int count = 0;

  @override
  Widget build(BuildContext context) => Scaffold();
}

// widget.g.dart
part of "widget.dart";

partial class MyState {
  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }
}

Functional Widgets

// stateless_widget.dart
import "dart:code_gen";

/// Creates a StatelessWidget based on a function and its parameters. 
/// 
/// This macro will generate the widget with the name [widgetName]. 
class WidgetCreator extends FunctionMacro {
  final String widgetName;
  const WidgetCreator({required this.widgetName});

  @override
  String generate() => [
    // because FunctionMacro generates top-level code, we can create a class
    "class $widgetName extends StatelessWidget {",
      // The fields: 
      for (final Variable parameter in allParameters)
        "final ${parameter.type} ${parameter.name};",

      // The constructor: 
      "const $widgetName({",
      for (final Variable parameter in allParameters)
        "required this.${parameter.name}",
      "});",

      // The build method: 
      "@override",
      "Widget build(BuildContext context) {$functionBody}",
    "}"
  ].join("\n");
}

// widget.dart
import "stateless_widget.dart";

part "widget.g.dart";

@WidgetCreator(widgetName: "MyButton")
Widget buildButton() => ElevatedButton(
  child: Text(""),
  onPressed: () {},
);

void main() => runApp(
  MaterialApp(
    home: Scaffold(
      body: MyButton(title: "Fancy title")
    )
  )
);

// widget.g.dart
part of "widget.dart";

// Notice how this is not a partial class, but rather a regular class
class MyButton extends StatelessWidget {
  final String title;
  const MyButton({required this.title});
  
  @override
  Widget build(BuildContext context) {
    // Notice how the => was desugared.
    return ElevatedButton(
      child: Text(title),
      onPressed: () {},
    );
  }
}

Implementation

Since I'm not an expert on the Dart compiler, this proposal is targeted at the user-facing side of code generation. Anyway, I'm thinking that the compiler can parse user-code like normal. When it finds an annotation that extends Macro, it runs the macro's generate function and saves the output. Since more than one macro can be applied to any class/function (by stacking annotations), the compiler holds onto the output until it has generated all code for a given file. Then, it saves the generated code into a .g.dart file (creating partial classes when applicable), injects the part directive if needed, and compiles again. This process is repeated until all code is generated. Dart does not need to support incremental compilation for this to work: the compiler can simply quit and restart every time new code is generated, and eventually, the full code will be compiled. It may be slow, but only needs to happen when compiling a macro for the first time. Perhaps the compiler can hash or otherwise remember each macro so it can regenerate code when necessary.

More detailed discussions in #1578 and #1483 discuss how incremental/modular compilation can be incorporated into Dart.

This behavior should be shared by the analyzer so it can analyze the generated code. Thus, any generated code with errors (especially syntax errors) can be linted by the analyzer as soon as the macro is applied. Especially since generating strings as code is inherently unsound, this step is really important to catch type errors.

Syntax highlighting can be implemented as well, but is not a must if the analyzer is quick to scan the generated code. A special comment flag (like // highlight) may be needed.

How IDE's will implement "Go to definition" will entirely depend on how that is resolved for partial classes in general, but since these will be standard .g.dart files, I don't foresee any big issues.

cc from previous conversations: @mnordine, @jakemac53, @lrhn, @eernstg, @leafpetersen

FAQ

• Why do you use partial classes instead of extensions?

  Extensions are nice. They properly convey the idea of augmenting a class you didn't write. However, they have fundamental limitations:

  1. Extensions cannot define constructors. This means Foo.fromJson() is impossible
  2. Extensions cannot override members. This means that Disposer and Dataclass wouldn't be possible.
  3. Extensions cannot define static members.
  4. Extensions cannot add fields to classes.

  I experimented with mixins that solve some of those problems, but you can't declare a mixin on a class that mixes in said mixin, because it creates a recursive inheritance. Also, I want generated code to be an implementation detail -- if we use mixins, other libraries can import it and use it. Partial classes perfectly solve this by compiling all declarations of partial class Foo as if they were a single class Foo declaration.

• Why not use a keyword for macros?

  I toyed around with the idea of macro MyMacro (like mixin MyMixin) instead of class MyMacro extends ClassMacro. There are two big problems with this. The first is that it is not obvious what is expected of a macro. By extending a class, you can easily look up the class definition and check out the documentation for macros. The other problem is that if we distinguish between functions and classes, there's no easy way to say that with a macro keyword. By using regular classes, you can extend FunctionMacro and ClassMacro separately, and possibly more. This also means that regular users can write extensions on these macros if they want to build their own reflection helpers.

  Also, the idea of special behavior applying to an object and not a special syntax isn't new. The async/await keywords only apply to Futures, await for applies to Stream, int cannot be extended, etc.

• Can I restrict a macro to only apply to certain types?

  This is something I thought about briefly. I don't see any big problems with this, and it could let the macro use fields knowing that they will exist. There are two reasons I didn't really put much work into this. Because I use macros as regular class, you can't simply use on. Would we use generics on ClassMacro? I'm impartial to it, but we'd have to have a lint to check for it since there is no connection between annotations and generics. Obviously this wouldn't apply to FunctionMacro or anything else. The second reason was that I want to encourage using reflection in code generation instead of simply relying on certain fields existing. For example, Disposer would be safer to write with this feature, but instead I opted to use reflection, and as such created an annotation that the user can apply to each field they want to dispose. And by using @override in the generated code, the analyzer will detect when the macro is applied to a class that doesn't inherit a void dispose().

• Why just classes and functions? What about parameters, fields, top-level constants and anywhere an annotation is allowed?

  I couldn't think of a good example. Reply if you have one and we can think about it. There were two reasons why it's unusual: One, reflection is a big part of macros. If there's nothing to reflect on, maybe code generation is not the right way to approach it. Two, macros should be logically tied to a location in the human-written code. Classes and functions were the most obvious. It's not so obvious (to me anyway) what sort of code would be generated from a parameter in a function. I suppose you may be able to extend Macro directly (depending on how reflection is implemented) and apply that to any code entity you like.

• Aren't Strings error-prone?

  Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

I'd love to hear feedback on this, but let's try to keep the conversation relevant :)

[esDotDev]

This is really cool, I appreciate the lack of magic, and the highly readable code.

It looks like it would be quite easy to support some special cases in auto-dispose as well, like:

// Usage
@shouldDispose(useCancel: true)
Timer timer = Timer();
... 
// Macro
String getDisposeMethodForType(Type type){
   if(type is Timer || type is StreamSubscription || useCancel) return "cancel";
   return "dispose";
} 
...
if (field.annotation is ShouldDispose){
    "${field.name}.${getMethodForType(field.type)}();",
}

This example could also be made more realworld if it supports null aware operators. It should write out controller?.dispose() if it's nullable, controller.dispose() otherwise.

Another thing to consider is how refactoring will work. If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

It might be nicer for maintainability if a prefix-system were used here, like:

@WidgetCreator()
Widget createMyButton(){ // Some error could occur if create is not the prefix here
   ElevatedButton(
      child: Text(""),
      onPressed: () {},
    );
}

Where MyButton portion is a contract, that updates when either createMyButton or MyButton is renamed.

[cedvdb]

I like the syntax. Although:

• For the dispose example you might want to check if dispose exists on that annotated var, maybe not here, but i can see it being useful somewhere else
• Would it be possible not to have to write part "person.g.dart"; ?
• The .g file you mentioned are available only to the compiler and not actually created ? I don't use a specific runner library for that reason, it clutter the code base.

To be honnest this looks like a buider that is built in dart and runs before each compilation. I stand by my position that most use cases there could or even should be resolved another way, but if that means we get all those features all at once, so be it.

[Levi-Lesches]

@esDotDev you can make it even simpler!

class ShouldDispose {
  final String methodName;
  const ShouldDispose({this.methodName = "dispose"});
}
// ...
@shouldDispose(methodName: "cancel")
Timer timer = Timer();
// ...
if (field.annotation is ShouldDispose){
    "${field.name}.${field.annotation.methodName}();",
}

If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

Widget createMyButton(){  // Some error could occur if create is not the prefix here

Again, to reduce magic, I don't want to introduce any new mechanism if we don't have to. An annotation would be enough, and you only declare (and maintain) the name of the widget in one place: in the annotation. Keep in mind, not all macros create widgets, so you can't introduce a new mechanism just for that.

[Levi-Lesches]

@cedvdb

I wouldn't use partial keyword as I wish we had Partial like typescript does for Record.

Refer to #252, not my idea

For the dispose example you might want to check if dispose exists on that annotated var, maybe not here, but i can see it being useful somewhere else

Well, it was just an example -- plus, I still don't have concrete details on reflection yet (I need to take a deeper look at the analyzer package). An important point of this proposal is that the analyzer should generate code, not just the compiler. That means that as soon as you mark a field as @shouldDispose, the corresponding field.dispose(); line will be generated, and an error will be shown if necessary. Since generated code is inherently unsound, this is the best way IMO to catch type errors. We can go a step further and say that all analyzer output for foo.g.dart should be shown in foo.dart, to allow devs to catch these errors even quicker.

Would it be possible not to have to write part "person.g.dart"; ?

Yes, I think Dart should automatically inject the part directive into person.dart (if not already present) as part of the code-gen step.

The .g file you mentioned are available only to the compiler and not actually created ? I don't use a specific runner library for that reason, it clutter the code base.

No, these files are actually created in the filesystem, totally accessible to the user. I mean, they'll probably go in the .gitignore, but they should be totally readable/inspectable to the user ("as if they had written it by hand"). When I tested my code, I had person.dart and person.g.dart in the same folder. For cleanliness, I'm not sure if that's what we want, but however we do it, it should be very clear (and at least in the part directive) where the generated file lives.

To be honnest this looks like a buider that is built in dart and runs before each compilation. I stand by my position that most use cases there could or even should be resolved another way, but if that means we get all those features all at once, so be it.

Yeah, that's pretty much exactly what this is. Although, hopefully we can optimize it to only run when we need it. I agree that most problems can be solved effectively with OOP and more functions, but there are some cases where the boilerplate just gets too much. But I'll bring up another point that @esDotDev once made: Making a StatelessWidget is not only boilerplate, it also clutters the codebase. It's much harder to read than a simple function, and it's only useful because of how Flutter is implemented. So you can have smaller functions that get expanded into full Widgets, but only the functions need to be maintained and read.

Same idea with toJson(). When you add a new field to a class, the compiler warns if you didn't add it to the fromJson constructor. But it doesn't tell you to add it to toJson(), and if you forget, you can corrupt your database. It's much safer to just maintain the fields themselves and let code-gen take care of the rest.

[esDotDev]

@esDotDev you can make it even simpler! Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

IDE lets you right-click on any Class or Method, and rename it, it scans the entire codebase and does a "safe" find and replace for those names. It's basically a glorified find and replace, with some sort of analyzer ability. http://screens.gskinner.com/shawn/d3ATmjLVlP.mp4

The importance as it relates here, is that it would make it very easy for anyone, anywhere in the codebase, to rename MyButton What happens to the "MyButton" string at that pt?

@esDotDev you can make it even simpler!

I think having the class handle super common cases like Timer and StreamSubscription manually is better than forcing developers to remember to type a magic "cancel" string, but we're into implementation details there, and all cases look easy enough regardless.

[Levi-Lesches]

Remember, WidgetCreator() and Disposer were just examples to show how you would make macros -- anyone can write a macro do whatever they want. You can use Macro.sourceName to get the name of the function and implement your own createMyWidget() pattern, or use reflection (once we figure that out) to make a safe AutoDisposer. The point is that it should be easy for everyone to do, and that's what this proposal tries to address.

[esDotDev]

Totally, no need to get into the weeds on implementation details, other than to reveal use cases.

In this case would be interesting to see how type checks would work (in general), and how null aware code might be written (in general). But I guess this is mostly a feature of the yet-to-be-written analyzer?

[Levi-Lesches]

Type safety is something I've pretty much given up on with code-gen. As you probably noticed from my first proposal, I tried very hard to shoehorn it in. It was a Macro class which generated any function declared in normal Dart and had a special @generate annotation. That way the signature can be preserved and the analyzer could keep it type-safe. But there were many problems with this:

1. What if you don't know the name of the function in advance? (eg, custom getters/setters)
2. What if you don't know the type in advance? (eg, custom getters/setters)
3. What if you want to generate a constructor? (eg, Person.fromJson())
4. What if you want to generate a top-level function/class? (eg, WidgetCreator)

All these, (particularly number 2!), showed me that there was no good way to handle this. That's why my new proposal focuses more on how to effectively integrate with the analyzer/regular way of writing Dart code. If the analyzer can instantly react when you apply a macro to a class, then that's essentially free type-safety. In other words, yes there will be a lot of type errors, but they'll be "errors" similar to the missing token } errors you get when you're still writing the code -- they're both easily fixable.

But I guess this is mostly a feature of the yet-to-be-written analyzer?

I was referring to the analyzer that currently comes with Dart. I plan to integrate this proposal specifically with the anlayzer package on pub.dev -- once I have time to figure it out. Based on what I hear around this repo, that package is hard to work with, so hopefully code-gen can motivate the right people to clean it up.

[esDotDev]

Sorry I just meant null aware code from the perspective of generation, like:

"controller${field.canBeNull? "?" : ""}.dispose();", // Outputs controller.dispose() or controller?.dispose()

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

I really like the idea of instant code changes, and then just leaning on the compiler to flag errors. I'm still not sure who would be checking for errors on strings literals though... if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

[Levi-Lesches]

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

In my example, Variable.type is a standard Type variable. That's how dart:mirrors does it, I'm sure analyzer does it like that too. So yes, anything you can do today you can do with a macro.

if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

Okay let's go a little more into detail for your example:

// function.dart
part "function.g.dart";

@WidgetCreator(widgetName: "MyButton1")
Widget buildButton() => ElevatedButton(/* ... */);

// main.dart
import "function.dart";

void main() => runApp(MyButton1());

Now, let's say we change MyButton1 to MyButton2. Depending on where we make the change, we get two outcomes:

1. Change the parameter in function.dart: This means that function.g.dart now contains MyButton2, so all the references in main.dart are invalid. An error immediately pops up in main.dart. However, if you use "rename all" in your IDE (if you even can on a string), it should replace all instances, and there will be no error.

2. Change the name in main.dart: WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

All this can be summed up by saying that the code behaves to Dart as though it was written by a human. You as the developer still have to make sure you know what you're doing. If your IDE can help you, great. Otherwise, you just have to be a little careful. Or you can make WidgetCreator play nicely with your IDE.

As for the actual code gen, there are two "stages" of debugging generated code. The first is the macro writer, who writes the macro. The string literal itself, like you said, won't report any errors. But as soon as you put @MyMacro() on a class/function, it gets generated into Dart code, which the analyzer can inspect. Both the macro writer and the users who use it will be able to see these errors as if they had written the code by themselves.

[esDotDev]

2. Change the name in main.dart: WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

This is where refactoring with IDE comes into play. Rather than "rename in main.dart", you can think of it as "rename everywhere that class is referenced across the entire project". This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed. The macro would then see that missing class, and presumably just re-generate it, just like it did the first time "MyButton" was declared.

At a high level, the IDE needs to tell the code gen system: This is what I just changed, and then the code-gen system could respond, but I don't think that is in scope of what you're proposing... or maybe it is? If you were to receive an event from IDE that "ClassA" is now "ClassB", could the declaration @WidgetCreator(widgetName: "ClassA") not be automatically re-written to use "ClassB"?

[themisir]

It would be great if we could somehow eliminate use of part 'something.g.dart' and .g.dart files. Instead of placing generated files on project the preprocessor could put those files on somewhere else like '.dart_tool' folder or in memory like other languages does. Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build. I hope you get what I meant here.

[Levi-Lesches]

Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build

Presumably, the compiler would be able to detect if the macro or the affected code changes, and only re-generate the code if it has. For example, if you import a macro that someone else made, it's very unlikely to change. That way, code-gen becomes a set-and-forget process, and won't impact the build process in the long-term. Where the generated code ends up going won't affect this process.

An often-requested feature of code-gen (which I agree with) is that generated code should not only behave like hand-written code, but should be just as accessible to devs, one reason being it makes debugging simpler. Which means not only inspecting generated code, but also seeing analyzer output, using an IDE's "Go to definition" feature, etc. In that sense it would be better to have a regular file. Dart is remarkable in it's lack of "magic", and hiding code from devs would really disrupt that.

[Levi-Lesches]

@esDotDev

This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed.

Ah, here's where we were talking past each other -- you should never (even with current code-gen tools) edit a .g.dart file. You would change the annotation WidgetCreator(widgetName: "ClassA"), either by hand or with the "rename everywhere" feature. Then code-gen will automatically regenerate the widget, since it detected that the macro annotation changed. If I'm not mistaken, this is the choice that all code-gen tools have made -- can you edit a toJson() that was generated by json_serializable?

[esDotDev]

Ah I see. Looking at how functional_widget works, you can rename the Foo to Bar with IDE, and everything compiles fine. Once the code-generator runs again, it recreates the .g file, restoring Foo and deleting Bar which causes compile errors because the rest of the code is still referencing Bar.

This seems basically inline with what you'd expect, any changes to generated code are overwritten next time generator is run. A quality of life feature could potentially be built into the IDE to not allow refactor on classes that originate in a .g file.

[Jonas-Sander]

Aren't Strings error-prone?

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

  String generate() => 
    'String greet() => "This is a $sourceName class";';

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

You still have all the benefits of the strings but still a way to use a "real" API later / as another way.

(don't know if AST is the right word)

AST generate(CodeGenerator generator) {
 return generator.fromString('String greet() => "This is a $sourceName class";');
}

then you could also later or as another API add more of a method based approach:

return generator.createMethod(
            name: 'toString', // name of created method
            returnType: String,
            parameters: [ // matches parameter in body reference function
                context.mirror.className,
                context.mirror.fields,
                Parameter(type: int, value: someParam), // just for API demonstration
            ],
            body: toJsonBody, // function for reference
        );

[Jonas-Sander]

@Levi-Lesches Is there any way you could fix the small typo in the title? 😁 ❤️

Code genertaion

[Levi-Lesches]

@Jonas-Sander:

Code genertaion

Sigh, no matter how much you review something mistakes always slip through! Fixed it, thanks.

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

Well, two points on this: First, the way this proposal is set up, the Dart team doesn't have to build in any new code to teach the compiler what the AST is, or how to generate code methodically. All it has to do is write strings to a file, which is easily done with dart:io. In fact, I tested my example code by simply plugging it into DartPad and putting the generated code in a new file. All the functionality of error-checking and type safety is done by simply generating the code and letter the analyzer do its thing. So adding in an API would add a massive amount of work to code-gen.

Secondly, after discussing it both in #1482 and #1507, I think many people came to the conclusion that having an API can end up making code-gen messier, not cleaner. Let's try fleshing out a full example

@override
Code generate(CodeGenerator generator) => generator.createMethod(
  name: 'toJson',
  // Dart doesn't support nullables or generics as `Type` values. So you can't have Map<String, dynamic>
  returnType: Map,  
  parameters: [  // are these going to be used in the generated function?
    context.mirror.className,
    context.mirror.fields,
  ],
  // what goes here? 
  // Is it the actual function or a function that generates code based on `className` and `fields`?
  // If it's the actual function, how do you use variables like the name of the class and return type?
  // If it's a function to generate the code, you might as well use that instead of an API
  body: toJsonBody, 
);

VS

@override
String generate() => "";  // start writing code immediately. 

Some other issues with an API, as mentioned in the above issues:

1. Such an API would have to keep up-to-date with new language features
2. If types are not known in advance, there is no way to keep the API type-safe
3. Understanding such code is harder than simply reading regular Dart code
4. An API has to be expressive enough to cover every single use-case. It essentially has to duplicate Dart
5. It has to be easy for all developers to use. Otherwise, we're back at square one.

In other words, I stand by my original comment:

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive.

If you have a specific use-case in mind that feels awkward with strings, please feel free to share.

[Jonas-Sander]

I completely understand your arguments "string vs AST".

I guess for me it just feels weird to have something as "low level" as a string directly as a return value from the generate method.

My argument is just that it would be more extensible to create the code via string-templating with something like my example of a CodeGenerator class. This API based approach was just an example to show that in this way you could still extend how you generate code without breaking old users.

Also what about some parameters to tune the code generation?

return generator.fromString(`...`, someTemplatingOption: true);

// Or
generator.setSomeOption(false);
generator.appendString(`...`);
generator.appendString(`...`);
return generator.code;

My point is that it AST was just some example of another Api that could thus added later in time without make breaking changes.

Is there a way to also set options by overriding getter in Macro class already? Yes.

May there be use cases where named arguments or methods on the CodeGenerator class make more sense? I don't know. Maybe you have some thoughts.

In the end my biggest concern was really just my gut feeling having something like a String given back. I know it makes sense if you only use string templating to write code. Still there may be some benefits to an approach as described.

[renggli]

I am surprised that all the discussions around Dart code generation remains at the level of string concatenation that is later parsed by the compiler. This seems unnecessarily low-level and detached from the actual language 👎🏻

Lisp and Scheme have a really nice concept around self-evaluating forms and quoting. I realize this is harder to adopt for a more complicated language like Dart; but I think it is worth to investigate and has successfully been done for other languages such as R, Rust, Scala, Smalltalk, Haskell, OCaml, C#, Julia, ...

[Levi-Lesches]

@Jonas-Sander, I'd agree with you if there was some someTemplatingOption that made sense, but I haven't been able to think of any. If you're talking about a parameter which changes the generated code, add a parameter to your macro's constructor. If you're talking about a parameter to change the formatting of the generated code, I would actually prefer that dart format be run automatically instead.

@renggli, based on that Wikipedia section you linked, it seems like "quoting" is just... string concatenation?

Both Common Lisp and Scheme also support the backquote operator (termed quasiquote in Scheme), entered with the ` character (grave accent). This is almost the same as the plain quote, except it allows expressions to be evaluated and their values interpolated into a quoted list with the comma , unquote and comma-at ,@ splice operators. If the variable snue has the value (bar baz) then `(foo ,snue) evaluates to (foo (bar baz)), while `(foo ,@snue) evaluates to (foo bar baz). The backquote is most often used in defining macro expansions.

I haven't used Lisp, so tell me if I'm wrong, but when you define x as (y z), Lisp is saving not only the values of y and z, but also their names. Later, when you use ` and `@, Lisp replaces x with the string y z, to be interpreted/evaluated by Lisp later.

I can understand to many that generating strings can feel wrong, but really everything is a string to begin with. When you write code in a .dart file, the compiler reads that as a string. No matter what form of generation you use -- API, strings, something like Lisp -- identifiers in your code are strings and have to be interpolated into your generated code somehow. Reflection is one way of understanding those strings at a higher level, but ultimately, generating strings as code is no different than typing by hand, which is what I was going for here.

Take the linked C# article:

Expression<Func<int, int>> doubleExpr = x => x * 2;

var substitutions = new Dictionary<ParameterExpression, Expression>();
substitutions[doubleExpr.Parameters[0]] = Expression.Add(
        Expression.Constant(3),
        Expression.Constant(4));

var rewritten = new ParameterReplacer(substitutions).Visit(doubleExpr.Body);
// (3 + 4) * 2

That's equivalent to:

String add(int a, int b) => "a + b";
String times2(String expression) => "($expression) * 2";
String generate() => double(add(3, 4));  // (3 + 4) * 2

The only difference is that there needs to be a way to convert strings back to code -- that's why this proposal uses .g.dart files.

There are also problems with using actual reflection (like tree-shaking) and type safety. While I'm not part of the Dart team, they've spoken up numerous times saying something like this would be incompatible with Dart today, or at least, very very hard. This proposal focuses on easing that strain to make code-gen more compatible with Dart and easier to implement, by treating code as strings, like humans would. Maybe this way, the Dart team may pick this up sooner rather than later. I'll quote @eernstg from #592:

We have discussed adding support for various kinds of static meta-programming to Dart, and it would be so cool, but it's a large feature and we don't have concrete proposals.

Now all that being said, I can certainly understand that this isn't going to satisfy a lot of people, and I encourage those who want to to try to find a way to fit code-gen more "natively" into Dart, but I would ask that it be in a new issue, because I consider it out-of-scope for this proposal. From Dart's perspective, they really are two entirely different problems -- generating strings and writing files is easily doable today, whereas full langauge support for metaprogramming would be a much greater effort.

[Levi-Lesches]

@tatumizer, based on your experience, do you feel this proposal would play nicely with the code you've written so far? Would it have made any parts of it easier? Harder?

[esDotDev]

This approach is really hard to argue with because:

1. The important use cases in Flutter are small, and this addresses them all easily, and dart isn't really used outside Flutter.
2. It requires very little work on the language side, so it will come to market very quickly
3. Because it's string-based, there is very little maintenance to be done as the language grows and changes

So in the end, the argument for something more complex doesn't seem to justify it's existence. It would not open up any new use cases we can't already do, it would take much longer to get developed, it would likely have bugs for many mths or yrs of it's existence, etc etc all the fun stuff that comes with an order of magnitude more complexity. And for what? Compile safety on snippets that most developers never see, and are virtually never modified? It's not worth it.

Especially when you consider the test/debug flow of the proposed solution, it has instant string output, so as you are typing and saving, you could see the new code being generated, it would be extremely easy to debug, and the compiler would flag errors in realtime as you're hitting save.

[Levi-Lesches]

(e.g. if my class is declared as a template, the compiler should allow methods without bodies (this can be achieved by declaring them "external", but maybe it can be made the default for templates)

BTW, as defined in #252, partial classes won't allow you to have empty bodies. In a case where you don't have the body yet, you just don't include the header either, and define both later.

I can implement them with no framework in 100 lines of code or less based on mirrors and strings, and I can do it rather quickly

Yep, that's basically the point. This gets rid of the need for mirrors, since mirrors aren't really supported everywhere and have issues with runtime.

Trivial examples don't provide much material for generalization.

But then if you get too complex, you leave the field of code gen and end up back in Dart. Consider your example: insert a print("Starting function X"); and print("Finished function X") in every function whose name starts with "foo". Instead, you can do this:

T log<T>(T Function() func, String name) {
  print("Starting function $name");
  final T value = func();
  print("Finished function $name");
  return value;
}

String myFunc() => "Hello, World!";
String functionWithArguments(String name) => "Hello, $name";

void main() {
  final String message1 = log(myFunc, "Basic message");
  final String message2 = log(() => functionWithArguments("John Doe"));
}

With just a few lines to define log, we're able to wrap any function with it, with some benefits:

• We no longer restrict the function to start with foo
• We can now print any message, not just the name of the function
• We can choose when we want to log. Consider:

void main(List<String> args) {
  final String Function() getMessage = args.contains("--debug")
    ? () => log(myFunc, "Basic message") : myFunc;
  print(getMessage();
}

In fact, I would argue that only simple cases of just "I have to type the same thing again and again" should be handled by code-gen, and everything else with regular Dart.

[esDotDev]

A "typical" use case is just a gap in the language that many developers want plugged. They will be things currently served by code-gen right now. I believe these are pretty exhaustive: https://github.com/dart-lang/language/issues/1458#issuecomment-815155034, comes down to maybe 6 or 7 concrete use cases that Flutter devs would love to have today. (Maybe there are some on the pure dart side we're missing?)

I'm sure once we get used to Macros many other common use cases will emerge, but I really hope this doesn't get over-engineered. If it can be simple, let it be simple.

You wouldn't see code-gen doing the example described above because we can use a Mixin, Util Method or an Extension to add methods to things, and it would make no sense to generate code when good compiler enforced language constructs will do. Code-gen is about filling gaps, or reducing repetition, not replacing language features.

[Levi-Lesches]

Sorry, I meant that in some cases where someone would be tempted to use mirrors, like automatic toJson and fromJson, now they can use code-gen instead.

[lrhn]

@tatumizer That sounds like a job for aspect oriented programming, not code generation.

You want to figure out which functions starting with foo in package:something are called, and how often. That's a cross cutting concern, so you define a join point matcher targeting those (with some fancy pattern syntax) and put an "around" advice on each matching target, allowing you to wrap all invocations of those methods.

Worked pretty well for Java back when I used it some 15 years ago. Not exactly new technology 😁 .

Code generation is usually more about creating new classes (or other types) or methods, ones which cannot be simply inherited or mixed in by the existing language. It's parameterized or abstracted declarations, rather than instrumentation of existing code.

(Obviously a general "I can rewrite anything" framework can do anything to a program. It's a very big gun, and if you can do anything, doing something specific can get harder. A more specialized framework can help you with what it's specialized for.)

[themisir]

Not sure what you mean by that. Everywhere? Where do you need them to be supported? On Android cellphone? Why? For code generation, we need them to be supported only in one place: in the VM. Do you need them in javascript or smart watch?

Yes exactly, we'll only need mirrors during compile time to work with the code.

This might increase build time since the code will have to be parsed twice (before and after the code generation). But we might reduce build time by placing "code generation" part after code parsing (where expression trees are generated). Instead of generating code we could manipulate expression tree to include our generated code or even do some fancy stuff like enumerating all of the methods that name starts with "foo*" and add an expression that will log "foo method is called".

Also it will give another advantages since "expression tree modifying macros" will work faster (cause it doesn't have to be parsed multiple times) than regular string code generators, we might get advantage on editor support. The analyzer might execute macros and give intelligence to editors about changed code. Also it will eliminate creepy '.g.dart' files in project directory.

Would this will make it harder to introduce meta programming?

• Yes but it will worth it. I can do lots of cool things with reflection in C# currently, and I would love to use same power on dart.

Wouldn't this make using meta programming harder?

• Yes of course. But not everybody have to use meta programming, you'll probably be using exists macros more than creating your own macros. With great power comes great responsibility.

What about maintainability?

• As I know dart compiler is already written in dart. So Dart team could separate expression tree domain to shared package so both macros and compiler will use same codebase for modifying expression trees.

[themisir]

Another idea, SDK might give us ability to modify built steps. So we can somehow execute our code between build steps (especially between tokenization and parsing or parsing and AOT/JIT) to make our own "meta programming" magic happen. This will not be modular as previous proposals but the communities might make their own meta programming patterns and later the the dart team might merge one of them to dart SDK, so Dart Team would have to do "less" work than designing modular meta programming interface.

This is all based on my assumption that Dart SDK is designed so it would support modularity.

[lrhn]

Yes exactly, we'll only need mirrors during compile time to work with the code.

What you want then is source mirrors. You don't need to reflect on run-time values, just source definitions. It'd be like an AST of the entire program, but without having to build the structure until you ask for it. It's also possible to have source mirrors which can't provide all the information (like not provide the source of function bodies in dependencies, only their signatures, which can then be supported by outlines for modular compilation).