Expose default values for parameters to macros?

[jakemac53]

The macro apis today do not give access to default values for parameters (or initializer expressions for variables either).

When augmenting a function today you are not allowed to specify default values, so we probably don't need them in that case. However, when overriding a method with a new declaration in a class (so you aren't augmenting an existing method/constructor, you are specifying a new one, which is an override), you do likely need access to the default values of the original parameters.

We could likely only provide these as Code objects and not values (the values might be user types, have unresolved identifiers, etc). But that should be sufficient as the use case is really just plopping in the same default value.

Concerns

I am concerned that in the case of unresolved identifiers (that will be produced in phase 2), we can't even always produce a valid augmentation library (at least in phase 1, although it would be challenging even in phase 2).

Consider a macro that runs in phase one, but it creates a type that extends some other type and overrides some members. Generally that would be hard to actually do since you can't introspect over members, but the macro could actually live on a member, something like this:

class A {
  @SomeMacro()
  void foo([int x = y]); // y is generated in phase 2!
}

// Generated in phase one, how do we refer to y properly if we don't know where it comes from?
class B extends A {
  void foo([int x = y])
}

Possibly we could just assume that any unresolved identifier will eventually exist in the current library, and not do any prefix. But that seems sketchy.

Other options

Alternatively, could we add a way of referring to the overridden default value? This would be generally useful, and would avoid the issue entirely. Maybe something like void foo({super.x}) or void foo({int x = super.default})?

[jakemac53]

cc @yanok @munificent

[lrhn]

If you can add a default value later, you can start by declaring

class B extends A {
  void foo([int x]);
}

in phase one, then add the = y in phase two.

I'm more interested in how you give access to an existing default value to the macro, so that it can use it elsewhere, like:

class C {
  C([int x = _someConstant]);
}

and you want to create a class:

class D extends C {
  C([int x = _someConstant]) : assert(x > 0), super(x);
}

in another library. You can't necessarily copy expressions from one context to another, and there isn't always a way to express a constant value in another library using Dart source at all.

[jakemac53]

If you can add a default value later

This is explicitly disallowed right now (augmentations in general cannot alter default values). For new methods in an augmentation if you define them in phase one (can only do this if also declaring a type), then if you do anything to them in phase two that is an augmentation of the one defined in phase one.

We could change how that works but it is much easier to think about things that way generally (each phases output is essentially an augmentation of the original library + previous phases augmentations).

I'm more interested in how you give access to an existing default value to the macro, so that it can use it elsewhere, like:

Good point regarding private identifiers, this is not something we can allow. It is a general problem for overrides though. Maybe this is another reason to go with a language feature, as it does solve a general problem that isn't macro specific?

[yanok]

+1 to what Lasse says. I think semantically default values are part of definition, not declaration. So maybe you should allow producing default values as part of definition phase and maybe this will simplify the resolution?

This of course means that function augmentation would have to support adding arguments' default parameters; for this use case it would be fine to restrict it to only support declarations with no definition, so we are not altering default values, just delaying their generation till the definition phase where they belong.

Private identifiers are painful in this respect, language feature would be nice. It doesn't help with my plans to rewrite Mockito, since I would need to grab v from A.m([int x = v]) while generating a class that doesn't extend A. A more general way to access default arguments, something like A.m.defaultArguments.x would help, but I don't really expect this to be ever implemented.

[jakemac53]

I think semantically default values are part of definition, not declaration. So maybe you should allow producing default values as part of definition phase and maybe this will simplify the resolution?

I consider them more a part of the API, they are an interesting/important part of the declaration itself. It is a grey area though, I could see an argument either way. But you generally can't change the default value and consider that non-breaking, as an example.

This of course means that function augmentation would have to support adding arguments' default parameters; for this use case it would be fine to restrict it to only support declarations with no definition, so we are not altering default values, just delaying their generation till the definition phase where they belong.

My main concern with this is just that it isn't something we can enforce statically - so it introduces a new category of runtime error that macro authors will have to manage (if they try to add a default value but one already exists).

I would much rather go with a language feature of some sort to avoid the situation entirely.

[jakemac53]

Also from a more concrete design perspective, the definition phase currently has the property that ordering does not matter, it can be fully concurrent (at least in terms of running the macros, their results must appear in a certain order still though to get a consistent augmentation).

If you can add default values, but only once, we lose that property. We also have to keep track during execution of macros whether or not a previous macro has added a default value, which adds an extra complication and cost that we don't have today. I don't think it provides enough value to be worth the complication or the runtime cost.

[lrhn]

Can an augmentation add a default value, if used by itself?

I'd personally expect the choice of everywhere to have a default value, or at least which default value, to be a late choice, like when you choose whether the function body is async or not. It's part of the function implementation, not its signature. Maybe having a non-nullable optional parameter screams "must have a default value", but that just means it'll be an error if you don't add one before the program must be complete, like other macro "full in the blanks".

[jakemac53]

Can an augmentation add a default value, if used by itself?

The general principle we have been following is that augmentations cannot affect the API of functions. You should be able to use them exactly as they appear in code, and not have magic extra parameters appear (and especially not have a parameter change types or something).

Whether that same logic should apply to default values is not is not very clear-cut, but the current decision is that yes it does. We do not allow adding default values at all via augmentation.

Personally I think that is the correct choice, because I do think they are a part of the API contract - at least sometimes. For instance if I have this function String substring(String original, [int length = 1]), clearly the default value of length is a part of the API contract. If it changed, lots of code would break.

Maybe having a non-nullable optional parameter screams "must have a default value", but that just means it'll be an error if you don't add one before the program must be complete, like other macro "full in the blanks".

I think this argument is sensible, a savvy enough reader should know that there must be a default value somewhere, and a smart enough IDE plugin could even show it to you inline as if it was there.

But, I don't think there is a lot of value in it. For hand written augmentations you can just as easily put it in the original function. For macros I see a lot of issues (listed in previous comments).

[scheglov]

Personally I think that is the correct choice, because I do think they are a part of the API contract - at least sometimes. For instance if I have this function String substring(String original, [int length = 1]), clearly the default value of length is a part of the API contract. If it changed, lots of code would break.

Isn't the breakage orthogonal to where the default value is? Is you change the "default" value here, clients also might break.

void f({int? a}) {
  a ??= 42;
  print(a);
}

[jakemac53]

Is you change the "default" value here, clients also might break.

That is a fair argument 🤷 . I am still hesitant about allowing macros to add default values though because it is difficult to enforce the "only allow it if it isn't already there" part. But we do already have some similar things, in the declarations phase. I would like to keep them at least constrained to that phase.

We could also just tell macro authors to use that pattern instead of regular default values.

[jakemac53]

cc @munificent thoughts?

[jakemac53]

https://github.com/dart-lang/language/issues/2269 looks like it would work as the language feature if we went that route

[jakemac53]

I do think I am pretty convinced this just needs to be a language feature.

In the short term though that does mean that mockito won't be a viable target for early macro testing, unfortunately.

[yanok]

I think there are two separate issues here: allowing to introspect default arguments and allowing to augment method definitions with default arguments.

Please correct me if I'm wrong, but it looks like the issues are around augmentation, introspection shouldn't be a big issue, right?

Also, do I understand it correctly, that problems appear only if one wants to add default values? If I try to emit a completely new method definition with some arguments having default values, that won't be a problem, even today?

If these two things are true, I don't think the default value thing makes it worse for "mockito on macros". The main blocker is still getting to introspect definitions that are known to be complete (because of being in another strongly connected component) during the early stages. If we could get this + introspection of arguments' defaults, mockito can happily emit everything in the types stage.

So, maybe mockito is not a viable target for macro testing, but for another reason: I can only think of one way to make it work, and this way won't exercise the phases interactions much (or at all), and you probably want that.

[jakemac53]

Please correct me if I'm wrong, but it looks like the issues are around augmentation, introspection shouldn't be a big issue, right?

Introspection is actually an issue, because of private variables. We can't give you a valid Code object that you could use in that case. We don't want to allow more detailed introspection than that (we have no API to represent an AST for expressions).

Also, do I understand it correctly, that problems appear only if one wants to add default values? If I try to emit a completely new method definition with some arguments having default values, that won't be a problem, even today?

Correct.

If these two things are true, I don't think the default value thing makes it worse for "mockito on macros".

Unfortunately the first is not true, I don't know how we work around the private variables issue. I also don't really want to expose this API at all if long term we have a different feature in mind - although if we only exposed it experimentally that might be ok (it could be marked deprecated from day one?). But it is still some wasted effort.

So, maybe mockito is not a viable target for macro testing, but for another reason: I can only think of one way to make it work, and this way won't exercise the phases interactions much (or at all), and you probably want that.

For early macros it would still be valuable, it would still exercise the other parts of the system. Also knowing that something like mockito can be done in the first phase is in itself valuable.

[yanok]

Introspection is actually an issue, because of private variables. We can't give you a valid Code object that you could use in that case. We don't want to allow more detailed introspection than that (we have no API to represent an AST for expressions).

Right, but you should probably be able to return Code or InvalidCode, right? At least for methods of classes that come from the "known-complete" libraries (other strongly connected component)? Alternatively mockito implementation could scan a Code object for private names (but I'd prefer to avoid that). Or we could just use the Code object and hope for the best. For experimentation sake it's probably fine, worst case the compiler would bark on the augmentation code. We could tell experimental mockito users: "If you see compiler complaining like this, you have to add that macro argument". It's not the UX I'd like to get in the end, but might be fine for first experiments.

I also don't really want to expose this API at all if long term we have a different feature in mind

Agreed. But it could probably be deprecated from day one as you say, or even not be a part of public API.

My point is it's probably too early to say "#2269 will just solve this", since this new language feature was not even approved yet. Are we sure it will eventually be in the language?

[jakemac53]

Right, but you should probably be able to return Code or InvalidCode, right? At least for methods of classes that come from the "known-complete" libraries (other strongly connected component)?

We could possibly do something like this, (probably throw if it can't be created, I wouldn't want a class like InvalidCode which implements Code to exist, as it would be easy to cause a failure later on which is harder to track down).

But it doesn't seem like a good long term solution so I hate to design and ship an API for a new language feature that we will have trouble removing later.

My point is it's probably too early to say "#2269 will just solve this", since this new language feature was not even approved yet. Are we sure it will eventually be in the language?

I would say today that feature is unlikely to ship unless the macro feature could make a strong argument for it existing.

But, I also think that mockito is pretty unique in its requirements here. Another feature that I think we will more likely ship is if constructs for argument lists, so you can optionally pass an argument. That would actually solve many use cases for this too.

[eernstg]

I would say today that feature is unlikely to ship

Well it's been around for about a year, but patterns and null safety were around for about a decade, and still went ahead. ;-)

Granted, it is a specialized mechanism: The need for denoting a default value generally only comes up when there is a need to do some kind of forwarding. So the question is basically whether we want support for safer/cleaner forwarding, and then how we'd do that. #2269 is one possible approach, and abstraction over actual argument lists (including the ability to pass-or-not-pass an actual argument programmatically) is another.

[pattobrien]

I have a use case detailed in #3611 that requires the computed value of any parameter's default, in order to expose the API contract of a given function to a generated ArgParser via a flag/option.

@cliCommand
void someFunction({
  String value = varValue,
}) {}

const varValue = 'foo';

// -- generates: --
final someFunctionParser = ArgParser()
  ..addOption(
    'value',
    defaultsTo: 'foo', 
    ...
  );

In this simplified example, the macro would create a command some-function with default value foo, rather than just varValue or unknown String. This would be impossible without evaluation access to the default Code object.

Introspection is actually an issue, because of private variables. We can't give you a valid Code object that you could use in that case.

I think if the macro user (function author in the above case) chooses to use a private variable as the default to a parameter, then their intent is clear and lack of introspection on that value would be okay. But for any public elements, introspecting the default value and evaluating the Code object would be expected, at least to some extent.

[lrhn]

What we can do depends on which requirements we need to satisfy. If we cannot set a default value in a macro generated augmentation file which could not be written by hand, then there is no way we can refer to a value whose computation requires accessing an inaccessible name.

If we ignore that, and pretend to be generating Kernel code, not source syntax, then nothing is impossible, and we could hypothetically just say "use that constant value" without giving any way to express the value using source.

That's what the compilers do today when they have forwarding factory constructors, or mixin application forwarding constructors, inherit the default value of the constructor they forward to. The compilers can and do allow "inferring" constants that cannot be expressed by user-written code in the same place.

But that would give macros the ability to do something you cannot (currently) do in any other way, which would probably quickly lead to a macro just for copying default values.

(I'd rather make it possible to programmatically call a function without a value, triggering the default value, without the combinatorial explosion when there are multiple optional parameters. Something like foo(if (test) value), which is equivalent to test ? foo(value) : foo(). And make it possible to detect whether a parameter was passed or not, like foo([late int x]) { ... } + some way to test if x is initialized yet. Then it shouldn't be necessary to know the default value of a function in order to forward to it.)

[pattobrien]

But that would give macros the ability to do something you cannot (currently) do in any other way, which would probably quickly lead to a macro just for copying default values.

I hear you, that's definitely not ideal. For calling the function, I agree that conditionally passing in arguments would be the best case forward. However there's a part of the use case I'm speaking of that has nothing to do with calling the function or augmenting it - it has more to do with capturing the entire function signature for use elsewhere in the program. In the case of cli-gen, that signature (including default value, doc comments) is printed out to the user when the my-app --help cli option is used.

Similarly, I've been working on a backend code-gen feature that analyzes annotated Dart functions to generate an OpenAPI schema of the entire API, which can then be served by the annotated shelf server app. Default value computation and doc comment introspection is just as necessary there.

I recognize that these are rather niche use cases, and possibly stretching the bounds of what macros are being built to solve, but they're real use cases that would otherwise require either a) continued use of build_runner or b) dedicated @DocComment('...') or @Default(42) parameter annotations to work around the limitation.. Obviously not the worst thing in the world, but not the best either :)

[munificent]

cc @munificent thoughts?

Sorry for the very long delay. Yeah, in general, we won't be able to take an expression in one context, paste into generated code in some other context and rely on that being able to work because of privacy and name resolution. As @lrhn says, we could add some magic mechanism that macros have access to in order to enable that to work.

But I'm inclined to interpret this as a signal that a language feature is worth having to "inherit" the default value from the overidden member.

I'm also OK with simply saying (for now, or perhaps forever) that macros just don't have access to default value expressions and macro authors have to work around that limitation. If that means that you can't have a macro generate an override of a method that has a default value, that's OK. I suspect that most macro-generated methods won't be overrides and of the ones that are, it's reasonable to simply limit them to methods that don't have optional parameters with non-obvious default values.

[GregoryConrad]

This seems like a case where it'd make sense to just adopt the long-awaited syntax like:

@Macro()
void myFn({int i = 123}) {}

// generated:
void myFn2({int? i}) {
  myFn(
    if (i != null) i: i,
  );
}

The above obviously doesn't help when you have a nullable parameter with a default value, but that could probably be done with the same copyWith workaround involving const _defaultParameter = DefaultParamObject(); that is seen everywhere today.

[rrousselGit]

Exposing default values to macros is definitely a requirement IMO.

I've already encountered cases where I have no solution to the problem I'm trying to solve because of it. We can't even use the @Default('someDefault') syntax, because the default value needs to respect import prefixes.

A @Default('<Model>[]') may actually need to be @Default('<prefix2.Model>[]'). And that's not a reliable solution.