Code generation (metaprogramming) proposal v2

[Levi-Lesches]

This is my second draft of a proposal for code generation ("code gen" from now on) through specialized classes called "macros". The issue where this is being discussed is #1482 and my first proposal is #1507.

The key idea for me in code generation is that generated code should function exactly as though it were written by hand. This means that code gen should be strictly an implementation detail -- invisible to others who import your code. This further strengthens the need for code gen to be simple and expressive, so that any code that could be written by hand can be replaced by code gen, and vice-versa.

Another critical point is that code generation should not modify user-written code in any way. For this reason, part and part of are used to separate the human-written code from the generated. Partial classes (#252) help facilitate this, and this proposal heavily relies on them.

Definition of a macro

A macro is a class that's responsible for generating code. You can make a macro by extending the built-in Macro superclass. There are two types of macros: those that augment classes, and those that generate top-level code from functions. The Macro class consists of the generate function and some helpful reflection getters. The base definition of a macro is as follows:

// to be defined in Dart. Say, "dart:code_gen";

/// A class to generate code during compilation.
abstract class Macro {
  /// Macros are applied as annotations, see below.
  const Macro();

  /// The name of the entity targeted by this macro.
  String get sourceName => "";

  /// Generates new code based on existing code. 
  String generate();
}

Here are class-based and function-based macros, with Variable representing basic reflection.

Variable

/// A example of how reflection will be used. 
/// 
/// This class applies to both parameters and fields.
class Variable<T> {
  /// The name of the variable
  final String name;

  /// Any annotation applied to the variable.
  final Object? annotation;

  /// Represents a variable.
  const Variable(this.name, [this.annotation]);

  /// The type of the variable.
  Type get type => T;
}

ClassMacro

/// A macro that is applied to a class. 
/// 
/// Generated code goes in a partial class with the same name as the source.
abstract class ClassMacro extends Macro {
  const ClassMacro();

  /// Fields with their respective types.
  List<Variable> get fields => [];

  /// The names of the fields 
  List<String> get fieldNames => [
    for (final Variable field in fields)
      field.name
  ];

  // More helpers as relevant...
}

FunctionMacro

/// A macro that is applied to a function.
/// 
/// Generates top-level code since functions can't be augmented. If you want 
/// to wrap a function, do it using regular Dart.
abstract class FunctionMacro {
  const FunctionMacro();

  /// The body of the function. 
  /// 
  /// => can be desugared into a "return" statement
  String get functionBody => "";

  /// The positional parameters of the function.
  List<Variable> get positionalParameters => [];

  /// The named parameters of the function.
  List<Variable> get namedParameters => [];

  /// All parameters -- positional, optional, and named.
  List<Variable> get allParameters => [];

  // More helpers as relevant...
}

That being said, I'm not an expert on reflection so this is not really the focus of this proposal. I do know enough to know that dart:mirrors cannot be used, since it is exclusively for runtime. We want something more like the analyzer package, which can statically analyze code without running it. Again, the point is that code gen should work exactly like a human would, and the analyzer can be thought of as a second pair of eyes (as opposed to dart:mirrors which is a different idea entirely). Depending on how reflection is implemented, we may need to restrict subclasses of Macro, since that's where all the initialization happens.

Using macros

To create a macro, simply extend ClassMacro or FunctionMacro and override the generate function. All macros will output code into a .g.dart file. The code generated by a ClassMacro will go in a partial class, whereas the code generated by a FunctionMacro will be top-level. Because the function outputs a String, List<String>.join() can be a clean way to generate many lines of code while keeping each line separate. dart format will be run on the generated code, so macros don't need to worry about indentation/general cleanliness. To apply a macro, use it like an annotation on a given class/function. Here is a minimal example:

// greeter.dart
import "dart:code_gen";  // imports the macro interface

class Greeter extends ClassMacro {
  const Greeter();  // a const constructor for use with annotations.

  /// Generates code to introduce itself. 
  /// 
  /// This function can use reflection to include relevant class fields. 
  @override
  String generate() => 
    'String greet() => "This is a $sourceName class";';
}

// person.dart 

// Dart should auto-generate this line if not already present 
part "person.g.dart";

/// A simple dataclass. 
/// 
/// This is the source class for the generator. So, [Macro.sourceName] will be 
/// "Person". The class is marked as `partial` to signal to Dart that the 
/// generated code should directly modify this class, as though they were 
/// declared together. See https://github.com/dart-lang/language/issues/252
@Greeter()  // applies the Greeter macro
partial class Person { }

void main() {
  print(Person().greet());
}

// person.g.dart
part of "person.dart";

partial class Person {
  String greet() => "This is a Person class";
}

They key points are that writing the generate() method felt a lot like writing the actual code (no code-building API) and the generated code can be used as though it were written by hand. Also, the generated code is kept completely separate from the user-written Person class.

Commonly requested examples:

The best way to analyze any new feature proposal is to see how it can impact existing code. Here are a few commonly-mentioned use-cases for code generation that many have a strong opinion on.

JSON serialization

// to_json.dart
import "dart:code_gen";

/// Generates a [fromJson] constructor and a [json] getter.
class JsonHelper extends ClassMacro {
  const JsonHelper();

  /// Adds a Foo.fromJson constructor. 
  String fromJson() => [
    "$sourceName.fromJson(Map<String, dynamic> json) : ",
    [   
      for (final String field in fieldNames)
        '$field = json ["$field"]',
    ].join(",\n"),
    ";"  // marks the constructor as having no body
  ].join("\n");

  /// Adds a Map<String, dynamic> getter
  String toJson() => [
    "Map<String, dynamic> get json => {",
    for (final Variable field in fields)
      '"${field.name}": ${field.name},',
    "};",
  ].join("\n");

  @override
  String generate() => [fromJson(), toJson()].join("\n"); 
}

// person.dart
import "to_json.dart";

part "person.g.dart";  // <-- added automatically the first time code-gen is run

// Input class: 
@JsonHelper()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

// Test: 
void main() {
  final Person person = Person(name: "Alice", age: 42);
  print(person.json);  // {name: Alice, age: 42}
}

// person.g.dart
part of "person.dart";

partial class Person {
  Person.fromJson(Map<String, dynamic> json) : 
    name = json ["name"],
    age = json ["age"];

  Map<String, dynamic> get json => {
    "name": name,
    "age": age,
  };
}

Dataclass methods

import "dart:code_gen";

/// Generates [==], [hashCode], [toString()], and [copyWith].
class Dataclass extends ClassMacro {
  const Dataclass();

  /// The famous copyWith method
  String copyWith() => [
    "$sourceName copyWith({",
    for (final Variable field in fields)
      "${field.type}? ${field.name},",
    "}) => $sourceName(",
    for (final String field in fieldNames)
      "$field: $field ?? this.$field,",
    ");"
  ].join("\n");

  /// Overrides the == operator to check if each field is equal
  String equals() => [
    "@override",
    "bool operator ==(Object other) => other is $sourceName",
      for (final String field in fieldNames)
        "&& $field == other.$field",
    ";"
  ].join("\n");

  /// Implements a hash code based on [toString()].
  /// 
  /// You can use more complex logic, but this is my simple version. It also
  /// shows that standard functions can be generated with macros. 
  /// 
  /// Make sure this is interpreted as [Macro.hash], not [Object.hashCode].
  String hash() => "@override\n"
    "int get hashCode => toString().hashCode;";

  /// Implements [toString()] by printing each field and the class name. 
  /// 
  /// Don't name it toString()
  String string() => [
    "@override",
    'String toString() => "$sourceName("',
    for (final String field in fieldNames)
      '"$field = \$$field, "',
    '")";'
  ].join("\n");

  @override
  String generate() => [
    equals(),
    hash(),
    string(),
    copyWith(),
  ].join("\n");
}

// person.dart
import "dataclass.dart";

part "person.g.dart";

@Dataclass()
partial class Person {
  final String name;
  final int age;
  const Person({required this.name, required this.age});
}

void main() {
  Person alice = Person(name: "Alice", age: 42);
  print(alice.hashCode);
  print(alice.toString());
  final Person alice2 = alice.copyWith();
  final Person bob = alice.copyWith(name: "Bob");
  if (alice == alice2 && alice != bob) {
    print("Equals operator works");
  }
}

// person.g.dart
part of "person.dart";

partial class Person {
  @override
  bool operator ==(Object other) => other is Person 
    && name == other.name 
    && age == other.age;
  
  @override
  int get hashCode => toString().hashCode;
  
  @override
  String toString() => "Person("
    "name = $name, "
    "age = $age, "
  ")";

  Person copyWith({
    String? name,
    int? age
  }) => Person(
    name: name ?? this.name,
    age: age ?? this.age,
  );
}

Auto-dispose

// disposer.dart
import "dart:code_gen.dart";

/// An annotation to mark that a class should be disposed. 
class ShouldDispose { const ShouldDispose(); }
const shouldDispose = ShouldDispose();

/// Calls .dispose on all fields with the [ShouldDispose] annotation.
class Disposer extends ClassMacro {
  const Disposer();

  @override
  String generate() => [
    "@override",
    "void dispose() {",
      for (final Variable field in fields)
        if (field.annotation is ShouldDispose)
          "${field.name}.dispose();",
      "super.dispose();",
    "}",
  ].join("\n");
}

// widget.dart
import "disposer.dart";

part "widget.g.dart";  // <-- injected by code gen 

class MyWidget extends StatefulWidget {
  @override
  MyState createState() => MyState();
}

@Disposer()
partial class MyState extends State<MyWidget> {
  @shouldDispose
  TextEditingController controller = TextEditingController();

  // Not included in the dispose function
  int count = 0;

  @override
  Widget build(BuildContext context) => Scaffold();
}

// widget.g.dart
part of "widget.dart";

partial class MyState {
  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }
}

Functional Widgets

// stateless_widget.dart
import "dart:code_gen";

/// Creates a StatelessWidget based on a function and its parameters. 
/// 
/// This macro will generate the widget with the name [widgetName]. 
class WidgetCreator extends FunctionMacro {
  final String widgetName;
  const WidgetCreator({required this.widgetName});

  @override
  String generate() => [
    // because FunctionMacro generates top-level code, we can create a class
    "class $widgetName extends StatelessWidget {",
      // The fields: 
      for (final Variable parameter in allParameters)
        "final ${parameter.type} ${parameter.name};",

      // The constructor: 
      "const $widgetName({",
      for (final Variable parameter in allParameters)
        "required this.${parameter.name}",
      "});",

      // The build method: 
      "@override",
      "Widget build(BuildContext context) {$functionBody}",
    "}"
  ].join("\n");
}

// widget.dart
import "stateless_widget.dart";

part "widget.g.dart";

@WidgetCreator(widgetName: "MyButton")
Widget buildButton() => ElevatedButton(
  child: Text(""),
  onPressed: () {},
);

void main() => runApp(
  MaterialApp(
    home: Scaffold(
      body: MyButton(title: "Fancy title")
    )
  )
);

// widget.g.dart
part of "widget.dart";

// Notice how this is not a partial class, but rather a regular class
class MyButton extends StatelessWidget {
  final String title;
  const MyButton({required this.title});
  
  @override
  Widget build(BuildContext context) {
    // Notice how the => was desugared.
    return ElevatedButton(
      child: Text(title),
      onPressed: () {},
    );
  }
}

Implementation

Since I'm not an expert on the Dart compiler, this proposal is targeted at the user-facing side of code generation. Anyway, I'm thinking that the compiler can parse user-code like normal. When it finds an annotation that extends Macro, it runs the macro's generate function and saves the output. Since more than one macro can be applied to any class/function (by stacking annotations), the compiler holds onto the output until it has generated all code for a given file. Then, it saves the generated code into a .g.dart file (creating partial classes when applicable), injects the part directive if needed, and compiles again. This process is repeated until all code is generated. Dart does not need to support incremental compilation for this to work: the compiler can simply quit and restart every time new code is generated, and eventually, the full code will be compiled. It may be slow, but only needs to happen when compiling a macro for the first time. Perhaps the compiler can hash or otherwise remember each macro so it can regenerate code when necessary.

More detailed discussions in #1578 and #1483 discuss how incremental/modular compilation can be incorporated into Dart.

This behavior should be shared by the analyzer so it can analyze the generated code. Thus, any generated code with errors (especially syntax errors) can be linted by the analyzer as soon as the macro is applied. Especially since generating strings as code is inherently unsound, this step is really important to catch type errors.

Syntax highlighting can be implemented as well, but is not a must if the analyzer is quick to scan the generated code. A special comment flag (like // highlight) may be needed.

How IDE's will implement "Go to definition" will entirely depend on how that is resolved for partial classes in general, but since these will be standard .g.dart files, I don't foresee any big issues.

cc from previous conversations: @mnordine, @jakemac53, @lrhn, @eernstg, @leafpetersen

FAQ

• Why do you use partial classes instead of extensions?

  Extensions are nice. They properly convey the idea of augmenting a class you didn't write. However, they have fundamental limitations:

  1. Extensions cannot define constructors. This means Foo.fromJson() is impossible
  2. Extensions cannot override members. This means that Disposer and Dataclass wouldn't be possible.
  3. Extensions cannot define static members.
  4. Extensions cannot add fields to classes.

  I experimented with mixins that solve some of those problems, but you can't declare a mixin on a class that mixes in said mixin, because it creates a recursive inheritance. Also, I want generated code to be an implementation detail -- if we use mixins, other libraries can import it and use it. Partial classes perfectly solve this by compiling all declarations of partial class Foo as if they were a single class Foo declaration.

• Why not use a keyword for macros?

  I toyed around with the idea of macro MyMacro (like mixin MyMixin) instead of class MyMacro extends ClassMacro. There are two big problems with this. The first is that it is not obvious what is expected of a macro. By extending a class, you can easily look up the class definition and check out the documentation for macros. The other problem is that if we distinguish between functions and classes, there's no easy way to say that with a macro keyword. By using regular classes, you can extend FunctionMacro and ClassMacro separately, and possibly more. This also means that regular users can write extensions on these macros if they want to build their own reflection helpers.

  Also, the idea of special behavior applying to an object and not a special syntax isn't new. The async/await keywords only apply to Futures, await for applies to Stream, int cannot be extended, etc.

• Can I restrict a macro to only apply to certain types?

  This is something I thought about briefly. I don't see any big problems with this, and it could let the macro use fields knowing that they will exist. There are two reasons I didn't really put much work into this. Because I use macros as regular class, you can't simply use on. Would we use generics on ClassMacro? I'm impartial to it, but we'd have to have a lint to check for it since there is no connection between annotations and generics. Obviously this wouldn't apply to FunctionMacro or anything else. The second reason was that I want to encourage using reflection in code generation instead of simply relying on certain fields existing. For example, Disposer would be safer to write with this feature, but instead I opted to use reflection, and as such created an annotation that the user can apply to each field they want to dispose. And by using @override in the generated code, the analyzer will detect when the macro is applied to a class that doesn't inherit a void dispose().

• Why just classes and functions? What about parameters, fields, top-level constants and anywhere an annotation is allowed?

  I couldn't think of a good example. Reply if you have one and we can think about it. There were two reasons why it's unusual: One, reflection is a big part of macros. If there's nothing to reflect on, maybe code generation is not the right way to approach it. Two, macros should be logically tied to a location in the human-written code. Classes and functions were the most obvious. It's not so obvious (to me anyway) what sort of code would be generated from a parameter in a function. I suppose you may be able to extend Macro directly (depending on how reflection is implemented) and apply that to any code entity you like.

• Aren't Strings error-prone?

  Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

I'd love to hear feedback on this, but let's try to keep the conversation relevant :)