Static Metaprogramming

[jakemac53]

Metaprogramming refers to code that operates on other code as if it were data. It can take code in as parameters, reflect over it, inspect it, create it, modify it, and return it. Static metaprogramming means doing that work at compile-time, and typically modifying or adding to the program based on that work.

Today it is possible to do static metaprogramming completely outside of the language - using packages such as build_runner to generate code and using the analyzer apis for introspection. These separate tools however are not well integrated into the compilers or tools, and it adds a lot of complexity where this is done. It also tends to be slower than an integrated solution because it can't share any work with the compiler.

Sample Use Case - Data Classes

The most requested open language issue is to add data classes. A data class is essentially a regular Dart class that comes with an automatically provided constructor and implementations of ==, hashCode, and copyWith() (called copy() in Kotlin) methods based on the fields the user declares in the class.

The reason this is a language feature request is because there’s no way for a Dart library or framework to add data classes as a reusable mechanism. Again, this is because there isn’t any easily available abstraction that lets a Dart user express “given this set of fields, add these methods to the class”. The copyWith() method is particularly challenging because it’s not just the body of that method that depends on the surrounding class’s fields. The parameter list itself does too.

We could add data classes to the language, but that only satisfies users who want a nice syntax for that specific set of policies. What happens when users instead want a nice notation for classes that are deeply immutable, dependency-injected, observable, or differentiable? Sufficiently powerful static metaprogramming could let users define these policies in reusable abstractions and keep the slower-moving Dart language out of the fast-moving methodology business.

Design

See this intro doc for the general design direction we are exploring right now.

[rrousselGit]

Would static function composition be in the scope of this feature?

At the moment, higher-order functions in Dart are fairly limited since they require knowing the full prototype of the decorated function.

An example would be a debounce utility:

void Function() debouce(Duration duration, void Function() decorated) {
  Timer? timer;
  return () {
    timer?.cancel();
    timer = Timer(duration, () => decorated());
  };
}

which allows us to, instead of:

class Example {
  void doSomething() {

  }
}

write:

class Example {
  final doSomething = debounce(Duration(seconds: 1), () {
  
   });
}

but that comes with a few drawbacks:

• obvious readability decrease
• our debounce utility is not very reusable. It works only on void Function(), but we'd like it to work for all functions.

With static meta-programming, our debounce could inject code in the class at compilation, such that we could write:

class Example {
  @Debounce(Duration(seconds: 1))
  void doSomething() {
    print('doSomething');
  }
  
  @Debounce(Duration(seconds: 1))
  void doSomethingElse(int value, {String named}) {
    print('doSomethingElse $value named: $named');
  }
}

[jakemac53]

There is a delicate balance re: static function composition, but there are certainly many useful things that could be done with it. I think ultimately it is something we would like to support as long as we can make it obvious enough that this wrapping is happening.

The specific balance would be around user confusion - we have a guiding principle that we don't want to allow overwriting of code in order to ensure that programs keep their original meaning. There are a lot of useful things you could do by simply wrapping a function in some other function (some additional ones might include uniform exception handling, analytics reporting, argument validation, etc). Most of these things would not change the meaning really of the original function, but the code is being "changed" in some sense by being wrapped.

Ultimately my sense is this is something we should try to support though. I think the usefulness probably outweighs the potential for doing weird things.

[mateusfccp]

I like Lisp approach (in my opinion, the utmost language when it comes to meta-programming). Instead of defining a @Debounce or something alike, we would define new syntax that would simply expand to a regular method at compile-time. I don't know, however, how much complex is to make something like this considering Dart syntax.

[lrhn]

For something like debounce, a more aspect-like approach seems preferable. Say, if you could declaratively wrap a function body with some template code:

class Example {
  void doSomething() with debounce(Duration(seconds: 1)) {
    print('doSomething');
  }
  
  void doSomethingElse(int value, {String named}) with debounce(Duration(seconds: 1)) {
    print('doSomethingElse $value named: $named');
  }
}

template debounce<R>(Duration duration) on R Function {
  template final Stopwatch? sw;
  template late R result;
  if (sw != null && sw.elapsed < duration) {
    return result;
  } else {
    (sw ??= Stopwatch()..start()).reset();
    return result = super;
  }
}

This defines a "function template" (really, a kind of function mixin) which can be applied to other functions. It cannot change the signature of the function, but it can access arguments (by forwarding them as with templateName(arg)), and it can do things before and after the original body is run. The template variables are per-template instantiation variables (just as we could declare static variables inside normal functions).

(Maybe we just need AspectD for Dart.)

[rrousselGit]

It cannot change the signature of the function, but it can access arguments

But an important part of function composition is also the ability to inject parameters and ask for more parameters.

For example, a good candidate is functional stateless-widgets, to add a key parameter to the prototype and inject a context parameter. This means the user would define:

@statelessWidget
Widget example(BuildContext context, {required String name}) {
  return Text(name);
}

and the resulting prototype after composition would be:

Widget Function({Key? key, required String name})

where the final code would be:

class _Example extends StatelessWidget {
  Example({Key? key, required String name}): super(key: key);

  final String name;

  @override
  Widget build(BuildContext) => originalExampleFunction(context, name: name);
}

Widget example({Key? key, required String name}) {
  return _Example(key: key, name: name);
}

[jakemac53]

I definitely agree we don't want to allow for changing the signature of the function from what was written. I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want. The original function might be private in that case.

[rrousselGit]

I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want

That's what functional_widget does, but the consequence is that the developer experience is pretty bad.

A major issue is that it breaks the "go to definition" functionality because instead of being redirected to their function, users are redirected to the generated code

It also causes a lot of confusion around naming. Because it's common to want to have control on whether the generated class/function is public or private, but the original function to always be private.

By modifying the prototype instead, this gives more control to users over the name of the generated functions.

[jakemac53]

Allowing the signature to be modified has a lot of disadvantages as well. I think its probably worse to see a function which is written to have a totally different signature than it actually has, than to be navigated to a generated function (which you can then follow through to the real one). You can potentially blackbox those functions in the debugger as well so it skips right to the real one if you are stepping through.

[bouraine]

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

[mateusfccp]

@bouraine

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

Yes.

[jakemac53]

@tatumizer This issue is just for the general problem of static metaprogramming. What you describe would be one possible solution to it, although we are trying to avoid exposing a full AST api because that can make it hard to evolve the language in the future. See https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md for an intro into the general design direction we are thinking of here which I think is not necessarily so far off from what you describe (although the mechanics are different).

[idkq]

Great intro & docs.

Hopefully we'll stay (far far) away from annotations to develop/work with static meta programming?!

[jakemac53]

so it looks like copyWith is seen as a crown jewel of the upcoming facility

The main reason we use this as an example is its well understood by many people, and it is also actually particularly demanding in terms of features to actually implement due to the public api itself needing to be generated :).

The language needs some mechanism of dealing with default values, which has been a showstopper in dart from day one.

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

[rrousselGit]

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

freezed supports this, but only because it relies on factory constructors and interface to hide the internals of copyWith (that is in fact a copyWith({Object? name = _internalDefault}))

[jakemac53]

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

Right, this is what I was describing which null safety actually does fix at least partially. You can make the parameter non-nullable (with a default), and then null can no longer be passed at all. Wrapping functions are required to copy the default value, basically it forces you to explicitly handle this does cause some extra boilerplate but is safe.

For nullable parameters you still can't differentiate (at least in the function wrapping case, if they don't provide a default as well)

[idkq]

Metaprogramming is a broad topic. How to rationalize? We should start with what gives the best bang for buck (based on use cases).

Draft topics for meta programming 'output' code:

1. Methods
2. Classes (shell)
3. Class members
4. Types
5. Enums
6. Statements (?)
7. Mixins (?)
8. Generics (?)

Also on output code:

Be able to visualize in some way the code generated into your program, at development time (https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md#usability)

Would be great if this could work without saving the file, a IDE-like syntax (hidden) code running continuously if syntax is valid. I refuse to use build_runner's watch

[porfirioribeiro]

Metaprograming opens doors to many nice features For the data class thing, this is something i miss from Kotlin. When i used Java we used @Data / @Value from Lombok that was some sort of generator, i guess having something like this would be enough for the data class's

Other language that does a great job at implementing macros is Haxe you can use Haxe language to define macros

I guess there are many challenges to implement this.

[ykmnkmi]

can we extend classes with analyzer plugin? can we use external and patch like patches in sdk libraries for extending classes? plugins for CFE?

[escamoteur]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

[jodinathan]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

I agree with this. I like the idea of meta programming as long as it doesn't remove how readable and maintainable a Dart code is.

[idkq]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

[leafpetersen]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

[jodinathan]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

In my honest opinion: things must be obvious, not magical. Every time I have to read or develop in PHP, JS or C with preprocessors etc... I just hate it. Too many magical stuff that you just can't read or debug easily. Dart is the opposite of that without being boring as hell as Java. In fact, there was a time that some Dart packages used to implement the noSuchMethod to create magical methods. Gee, what a pain. Meta programming could be the next Dart transformers if it takes the glittering magical road.

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

^ this

[esDotDev]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity.

But there is nothing beautiful about writing data classes or running complicated and and slow code-generation tools.

I'm hoping this can lead to more simplicity not less. Vast mounds of code will be removed from our visible classes. StatefulWidget can maybe just go away? (compiler can run the split macro before it builds?). Things can be auto-disposed. Seems like this could hit a lot of pain points, not just data classes and serialization..

[safasofuoglu]

Since dart currently offers code generation for similar jobs-to-be-done, I'd suggest evaluating potential concerns with that consideration:

• Metaprogramming is not simple/obvious - can it be made at least as simple/obvious as codegen through tooling?
• Metaprogramming will be abused - Is there a reason to think it will be abused more than codegen? (potentially, if it provides better ergonomics)

On the other hand, besides being an upgrade from codegen for developers, metaprogramming could provide healthier means for language evolution beyond getting data classes done. Quoting Bryan Cantrill:

Another advantage of macros: they are so flexible and powerful that they allow for effective experimentation. For example, the propagation operator that I love so much actually started life as a try! macro; that this macro was being used ubiquitously (and successfully) allowed a language-based solution to be considered. Languages can be (and have been!) ruined by too much experimentation happening in the language rather than in how it’s used; through its rich macros, it seems that Rust can enable the core of the language to remain smaller — and to make sure that when it expands, it is for the right reasons and in the right way. http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/

PS @jakemac53 the observable link leads to a private google doc.

[insinfo]

this would be fantastic if it allowed, the longed-for serialization for JSON natively without the need for manual code generation or reflection in time of execution

Today practically all applications depend on serialization for JSON, a modern language like dart should already have a form of native serialization in the language, being obliged to use manual codegen or typing serialization manually is something very unpleasant

[felixblaschke]

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

#ToStringMaker() // '#' indicates macro and will rewrite all code in next scope
class Person {
    String name;
    int age;
}

// [REWRITTEN CODE] => displayed readonly in IDE
// class Person {
//    String name;
//    int age;
//
//    toString() => 'Person(vorname:$name, age:$age)'
// }

class ToStringMaker extends Macro {

    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }

}

[idkq]

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

Interesting. I like the structure a lot, but not so much #ToStringMaker() (hashtag working as annotation). Syntax would be better if #ToStringMaker() could be incorporated into the class definition.

[jodinathan]

It is very interesting. I find decorators not solid to something that changes the code, so I would add the macro within the class definition. The Macro class could accept a generic abstract type to show users what methods, properties etc expect from the macro along with full documentation capabilities. The abstract type would also be used by the analyzer/compiler to validate the macro result.

class Person macros ToStringMaker() {
    String name;
    int age;
}

abstract class ToStringDefinition {
   // some method documentation
   String toString();
}

class ToStringMaker extends Macro<ToStringDefinition> {
    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }
}

[venkatd]

@jakemac53

This looks like a very useful feature!

What the Dart team picked the top 2-3 highest impact applications, built first-class implementations of them, and started with meta-programming itself as an implementation detail for the Dart language? This way you would all be free to refine meta-programming capabilities without making any public API promises to start.