proposal: spec: allow type parameters in methods

[mariomac]

According to the Type parameters proposal, it is not allowed to define type parameters in methods.

This limitation prevents to define functional-like stream processing primitives, e.g.:

func (si *stream[IN]) Map[OUT any](f func(IN) OUT) stream[OUT]

While I agree that these functional streams might be unefficient and Go is not designed to cover this kind of use cases, I would like to emphasize that Go adoption in stream processing pipelines (e.g. Kafka) is a fact. Allowing type parameters in methods would allow constructing DSLs that would greatly simplify some existing use cases.

Other potential use cases that would benefit from type paremeters in methods:

• DSLs for testing: Assert(actual).ToBe(expected)
• DSLs for mocking: On(obj.Sum).WithArgs(7, 8).ThenReturn(15)

---

Edited by @ianlancetaylor to add: for a summary of why this has not been approved, please see https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods .

[fzipp]

The document also explains what the problems are. So what are your solutions to these?

[zigo101]

This proposal is good to define an io.ImmutableWriter {Write(data byteview)(int, error)} interface: https://github.com/go101/go101/wiki/A-proposal-to-avoid-duplicating-underlying-bytes-when-using-strings-as-read-only-%5B%5Dbyte-arguments

[ianlancetaylor]

This proposal is a non-starter unless someone can explain how to implement it.

[deanveloper]

@ianlancetaylor from the generics proposal

Or, we could decide that parameterized methods do not, in fact, implement interfaces, but then it's much less clear why we need methods at all. If we disregard interfaces, any parameterized method can be implemented as a parameterized function.

I think this solution makes the most sense. They could then (under the hood) be treated a regular function. The reason why this would be useful is that methods do not only serve the purpose of implementing interfaces; methods also serve as a means of organization for functions that operate on particular structures.

It may be a bit of a challenge about how type-parameterized methods would appear in "reflect", though.

[zigo101]

The problem would be simpler if the parameter type possibility set is known at compile time,

It may be a bit of a challenge about how type-parameterized methods would appear in "reflect", though.

One new problem I'm aware of is there might be many methods have the same name for a certain type. So the Type.MethodByName might return a slice value (assume the parameter type possibility set is known at compile time). Any other new problems?

[AndrewHarrisSPU]

One new problem I'm aware of is there might be many methods have the same name for a certain type.

If we're ultimately talking about multiple dispatch, the languages that really cater towards this do a massive amount of overloading. One language I find fun and interesting with this style is Julia, where things like + or * or show have hundreds of overloads when booting the REPL. From a software engineering perspective, there are tradeoffs - I absolutely trust Go to compile long into the future, and to have fewer surprises about packages ... Remarkably and IMHO related to robustness, Go doesn't have programmers defining function overloads in source code - I'm not convinced generics should change this.

Particularly for stream-to-stream conversion, I do think that the List Transform example is useful. We have to provide a concrete T1 -> T2 conversion function, but in a sense we have to figure out how to convert T1 to T2 in any kind of system.

I think it's also often possible to have a higher-order function that generates conversion functions, while more specialized conversion functions are also naturally expressible in Go. Example: a very generic color conversion API might specify an interface with ToRGB() and FromRGB() methods, and this can go pretty far. We can express 8-bit to 16-bit RGB conversion here through the interfaces, the same as e.g. HSV or LAB conversions, but there's a faster bit-shifting path. With a sense of a generic default, something like bufio.Scanner seems plausible - where the default just works, but we can optionally provide a better color conversion the same way we can provide a different SplitFunc.

[DeedleFake]

@deanveloper

Even just that would allow for, for example, an iterator implementation, though it does require wrapping it in another type because of the lack of extension functions:

type Nexter[T any] interface {
  Next() (T, bool)
}

type NextFunc[T any] func() (T, bool)

func (n NextFunc[T]) Next() (T, bool) {
  return n()
}

type Iter[T any, N Nexter[T]] struct {
  next N
}

func New[T any, N Nexter[T]](next) Iter[T, N] {
  return Iter[T, N]{next: next}
}

func (iter Iter[T, N]) Map[R any](f func(T) R) Iter[R, NextFunc[R]] {
  return New(NextFunc[R](func() (r R, ok bool) {
    v, ok := iter.next.Next()
    if !ok {
      return r, false
    }
    return f(v), true
  })
}

// And so on.

Usage is still awkward without a short-form function literal syntax, though, unfortunately:

s := someIter.Filter(func(v int) bool { return v > 0 }).Map(func(v int) string { return strconv.FormatInt(v, 10) }).Slice()
// vs.
s := someIter.Filter(func(v) -> v > 0).Map(func(v) -> strconv.FormatInt(v, 10)).Slice()

[batara666]

This will add so much complexity

[deanveloper]

@batara666 can you explain why? adding type parameters to methods doesn’t seem like it’d add that much complexity to me.

[Merovius]

I think before we can think about if and how to do this, we should first address the "no higher level abstraction" restriction of generics, i.e. the inability to pass around a generic type/function without instantiation. The reason is that if we allowed additional type parameters on methods, we'd also de-facto allow to pass around generic functions without instantiation:

type F struct{}

func (F) Call[T any] (v T) { /* … */ }

func main() {
    var f F // f is now de-facto an uninstantiated func[T any](T)
}

Therefore, to allow additional type-parameters on methods, we also have to answer how to pass around uninstantiated generic functions.

Moreover, if we'd allow passing around uninstantiated generic types/functions, we could already build the abstractions given as motivations in the proposal-text.

So, given that solving "no higher level abstraction" is a strictly easier problem to solve, while providing most of the benefit of solving "no additional type parameters on methods", it seems reasonable to block the latter on solving the former.

Lastly, I'd urge everyone to consider that these limitations where not left in the generics design by accident. If they where really that easy to solve, the solution would have been in the design to begin with. It will take some time to solve them.

[zigo101]

The "type" concept almost means a memory layout. If we could use the 1.18 constraint concept as general types, then many problems will be solved.

A value of a subset type could be passed/assigned to a superset type. A function with a superset type parameter could be used as a function with a subset type parameter.

For Go, the change would be too large. It is best to experiment the idea on a new language.

[AndrewHarrisSPU]

I was (far too) pleased (with myself) when I figured out this is possible: https://gotipplay.golang.org/p/1ixYAwxwVss

The part about lifting values into type system 'symbols' feels like a bit DIY and tricky, not sure there isn't something better here. I did feel like the dispatch() call is technically interesting. Inferring from the function/method supplied to dispatch() wasn't obvious to me at first. Without overloading methods, just different instantiations of the dispatch() function, it is plausible to arrive at the correct dispatch over a set of concrete implementations.

[ianlancetaylor]

I'm putting this proposal on hold until we have more familiarity with the current generics implementation.

[mariomac]

Playing with a new library that intensively uses generics, I provided an equivalence implementation between a Method and a Function:

https://github.com/mariomac/gostream/blob/bf84997953f02b94e28da0d6c4d38585d2677df2/stream/str_to_str.go#L5-L14

At the end, the difference is only where the parameter is placed (as a receiver, or as a first argument), but the function allows you map to a Stream of different type and with the method you can only generate streams of the same type.

With the code from the above link, I verified that this compiles:

type Mapper[IT, OT any] func(Stream[IT], func(IT)OT) Stream[OT]
var _ Mapper[int, float64] = Map[int, float64]

Simplifying, and obviating some internals from Go that I might ignore, I could see the generic Mapper type as a "single-method generic interface", that is implemented by the Map function, and it can be instantiated into a Mapper instance with concrete types.

In order to overcome the No parametrized methods issue pointed by @fzipp, from my partial view, I think that the example issue can be approached the same way as Java does: using interface{} behind the scenes and panic if the customer did a bad assignment (also the compiler could warn about the unsafe operation). Then for example the code from the example:

func CheckIdentity(v interface{}) {
	if vi, ok := v.(p2.HasIdentity); ok {
		if got := vi.Identity[int](0); got != 0 {
			panic(got)
		}
	}

Would be translated to something equivalent to:

func CheckIdentity(v interface{}) {
	if vi, ok := v.(p2.HasIdentity); ok {
		if got := vi.Identity(0).(int); got != 0 {
			panic(got)
		}
	}

Then the third line would panic if the v interface does not implement Identity[int]. The same way that Go does currently when you try to cast an identity{} reference to a wrong type.

In this case, we are translating the error check from the compile time to the runtime, but anyway this is what we actually have now if lack of parametrized methods forces us to continue using unsafe type castings.

[ianlancetaylor]

In your rewrite of CheckIdentity what do we gain by using a type parameter with the method? If the code is not type checked at compile time, we can just return an interface type, which already works today.

[deanveloper]

I think that the example issue can be approached the same way as Java does: using interface{} behind the scenes and panic if the customer did a bad assignment

This is a bad idea in my opinion - Type erasure is one of the most annoying limitations of generics in Java. I think it would go against the grain of the simplicity that Go aims for.

[mccolljr]

I don't really have a horse in this race, but I find this proposal interesting and wanted to put down my thoughts.

Based on what I see here: https://go.godbolt.org/z/1fz9s5W8x This:

func (x *SomeType) Blah() { /* ... */ }

And this:

func Blah(x *SomeType) { /* ... */ }

compile to nearly identical code.

If we have a type S:

type S struct {
    /* ... */
}

...and S has a method DoThing with a type parameter T:

func (s S) DoThing[T any](arg T) { /* ... */ }

...then we effectively have a generic function with the signature:

func DoThing[T any](s S, arg  T) { /* ... */ }

Of course, if we have a generic type G:

type G[T any] struct {
    /* ... */
}

...and G has a method DoStuff with a type parameter U:

func (g G[T]) DoStuff[U any](arg U) { /* ... */ }

...then we effectively have a generic function with the signature:

func DoStuff[T, U any](g G[T], arg U) { /* ... */ }

In order to use either of these "functions", all of the type parameters have to be known.

That means that in the case of S, the only way to refer to S.DoThing is to instantiate it: S.DoThing[int], S.DoThing[float64], etc. The same is true for G, with the additional requirement that G is also instantiated: G[int].DoThing[float64], etc.

Within this limited context, it seems to me like it wouldn't be a huge leap to allow type parameters on methods - it ends up referring to what is essentially a generic function, and we know things about generic functions:

1. They can't be used unless all type parameters are known statically at compile time, and
2. Each unique instantiation results in a unique function, semantically speaking (the actual implementation of course may choose to use a single function and internally use type switching, etc, etc)

The mechanism by which this interacts with interface definitions/implementations is less clear to me. Though, I think it is reasonable to say that a generic interface can't be implemented directly - it must be instantiated first. I'm not as sure of this, but it seems that it might also be true that an interface can only be implemented by a fully instantiated type.

Even in code like this:

type GenericInterface[T any] interface {
    Foo() T
}

type GenericStruct[T any] struct {
    Bar T
}

func (g GenericStruct[T]) Foo() T {
    return g.Bar
}

func MakeGeneric[U any]() GenericInterface[U] {
    return GenericStruct[U]{}
}

It seems like, within the context of MakeGeneric[T], we could consider both GenericInterface[T] and GenericStruct[T] to be instantiated with the some specific type T, which is the type value given to the type parameter U in MakeGeneric[U]. The determination that GenericStruct[T] implements GenericInterface[T] in this context is different from making a general statement that "GenericStruct[T] implements GenericInterface[T] for all T", which is what I would think of as "implementation without instantiation"

One area that seems complex is interfaces whose methods have type parameters.

For example, if we had:

type Mappable[T any] interface {
    Map[U any](func(T) U) []U
}

What would it mean to "instantiate" Mappable[T]? Can you use a type assertion such as blah.(Mappable[int]? If Mappable[T].Map had the signature Map[U comparable](func(T) U) []U, would a type with a method Map[U any](func(T) U) []U be treated as implementing Mappable[T]? This kind of interface seems to introduce a lot of ambiguity that would be difficult to resolve in a satisfactory manner.

It seems much simpler to disallow that kind of interface entirely, and just require something like:

type Mappable[T, U any] interface {
    Map(func(T) U) []U
}

I think that could still be just as useful, depending on how interface implementation is handled when the underlying type has methods with generic parameters.

As an example:

// Slice[T] provides slice operations over a slice of T values
type Slice[T any] []T

// Map[U] maps a Slice[T] to a Slice[U]
func (s Slice[T]) Map[U any](func (T) U) Slice[U]

type Mappable[T, U any] {
    Map(func (T) U) Slice[U]
}

// In order for this assignment to be valid:
// 1. Slice[int] must have a method named Map ✅
// 2. Slice[int].Map must have the same number of arguments ✅
// 3. Slice[int].Map must have the same number of returns ✅
// 4. Slice[int].Map must have the same types for each argument and return ???
var _ Mappable[int, float64] = Slice[int]{1,2,3}

It seems reasonable to me to say that Slice[int] implements Mappable[int, float64], since the method Map on Slice[int] can be instantiated & called with a U set to float64.

In this case, assuming that methods with type parameters are allowed, I would think the compiler could do something like:

1. Notice that Mappable[int, flloat64] is implemented for Slice[Int] when Slice[int].Map is insantiated with float64
2. Generate the code for that instantiation of Slice[int].Map, and
3. Use the pointer to that particular instantiation of Slice[int].Map in the vtable

If you're calling the method an the interface object, then you only have access to that one particular instantiation of the Slice[int].Map method. If use a type assertion to get back the original Slice[int] type, then you can of course call any number of Map variants on it, because the compiler knows what the concrete type is again.

To summarize:

Given that it is a feature of go that interface implementation can be tested at runtime via type assertions, reflection, etc, I don't see any way around banning generic methods on interface definitions. However, because methods are more or less sugar for functions, it seems to me it would be possible to allow generic parameters on methods of concrete types, and to allow these methods to participate in implementation of fully instantiated interface types.

[Merovius]

Given that it is a feature of go that interface implementation can be tested at runtime via type assertions, reflection, etc, I don't see any way around banning generic methods on interface definitions. However, because methods are more or less sugar for functions, it seems to me it would be possible to allow generic parameters on methods of concrete types, and to allow these methods to participate in implementation of fully instantiated interface types.

I don't think you are solving the problems from the design doc, though:

type IntFooer interface { Foo() int }
type StringFooer interface { Foo() string }
type X struct{}
func (X) Foo[T any]() T { return *new(T) }

func main() {
    var x X
    x.(StringFooer) // How does this work? Note that we can't use runtime code generation
    reflect.ValueOf(x).MethodByName("Foo").Type() // What is this type?
}

This fulfills all your criteria, it uses no parameterized interfaces and X is a concrete (non-parameterized) type. In particular, answering these questions here is the minimum required to make this feature useful.

It is very easy to look at the proposal text and think "this would be a useful feature to have, I obviously would like it in the language". But because it's such an obvious feature to put in, it would be great if people ask themselves why the Go team didn't put it in in the first place. Because there are reasons and these reasons need answering.

[alvaroloes]

I would like to drop an idea here which I think can be useful for the "type parameters in methods" topic. Maybe it has an obvious flaw I haven't seen or it has already been considered or discussed, but I couldn't find anything about it. Please, let me know if so.

With the current proposal, we can't have type parameters in methods but, couldn't we achieve the same effect if we put the type parameters in the type definition?

I mean, instead of doing this:

type Slice[T any] []T

func (s Slice[T]) Map[U any](func (T) U) Slice[U]

Do this (move the U type parameter from the method "Map" to the struct):

type Slice[T any, U any] []T

func (s Slice[T,U]) Map(func (T) U) Slice[U]

@Merovius Wouldn't this solve the issue you mentioned in the above comment? Your example would end up like this:

type IntFooer interface { Foo() int }
type StringFooer interface { Foo() string }
type X[T any] struct{}
func (X) Foo() T { return *new(T) }

func main() {
    var x X[int] // You are forced to specify the type parameter here with the current proposal. I guess it could be inferred it this were an initialization instead of a declaration only
    x.(StringFooer) // This would fail, as it doesn't conform to the interface
    x.(IntFooer) // This would pass
    reflect.ValueOf(x).MethodByName("Foo").Type() // "Foo(X) int"
}

I think this would work with the current proposal without changes.

As I said, I could be missing something obvious here. Let me know if that's the case.

[Merovius]

@alvaroloes

couldn't we achieve the same effect if we put the type parameters in the type definition?

You can easily do this, but it's not the same effect. People who want this specifically want the type-parameter of the method to be independent of the type itself, to implement higher-level abstractions.

[alvaroloes]

I see, thanks. After thinking it twice, I now see that what I propose would be very limiting as, after instantiating the type, you could not call the method with different type parameters (for example, call "map" with a function that return strings one time and then another time with a function that return ints).

All right, I know there was something obvious here. Thanks for the response!

[mccolljr]

I don't think you are solving the problems from the design doc, though:

I'm sure that's true, I probably would benefit from reviewing it again. To be fair, though, I wasn't trying to put together a concrete proposal - I understand why this is a complex topic and why it isn't in the first pass at generics, and why it may never be added to the language. I don't have a horse in this race beyond the fact that I find this interesting. My intent was to think out loud about what restrictions might make this more concretely approachable. Also, for what it's worth, I think that disallowing parameterized methods on interface types could be seen to addresses some of the problems put forth in the proposal.

type IntFooer interface { Foo() int }
type StringFooer interface { Foo() string }
type X struct{}
func (X) Foo[T any]() T { return *new(T) }

func main() {
    var x X
    x.(StringFooer) // How does this work? Note that we can't use runtime code generation
    reflect.ValueOf(x).MethodByName("Foo").Type() // What is this type?
}

[...] answering these questions here is the minimum required to make this feature useful.

I do not disagree. I feel as if you may have misinterpreted my intent. I am not saying "this is so easy, look at how we can do it" - I am saying "here is an interesting constraint that might make this more approachable, and which could perhaps be used as the basis for additional discussion"

I'm willing to brainstorm this, but again I am not proposing a concrete solution as much as attempting to provide a possible set of constraints for discussion. If that exercise shows that thia feature would too complex, that is a totally acceptable outcome in my opinion.

Obviously we cannot use runtime code generation, I don't recall proposing that nor do I think it is necessitated by anything said above. Given that, here are some possible (not exhaustive or comprehensive) directions the compiler could choose:

For x.(StringFooer)

• The program could encode the types that a generic method/function/etc has been instantiated with. This would allow x.(StringFooer) to correctly select the implementation of Foo that applies. Of course, if X.Foo is never explicitly instantiated with string, then it could be surprising to a user that this fails. Perhaps the cost of that potential confusion is unacceptable. This failure could of course be solved by adding var _ StringFooer = X{} somewhere in the code, and perhaps the panic message could indicate that the failure was due to uninstantiated generic methods rather than uninstantiated

• The compiler could generate a fallback implementation using interface{} or some minimum interface that it can use in these situations. In the case of type sets, the fallback could use a type switch. Perhaps if type switching on ~ types is implemented this becomes easier.

For reflect

• Similar to above, the compiler could generate metadata about which instantiation were generated and this could be introspectable from reflect. A public IsGeneric flag could be added to the descriptor for methods and calls to method values could validate the given types against the instantiation list to verify the proper code had been generated.

• Similar to above, the compiler could simply generate fallback implementations for generic methods (functions etc).

It is very easy to look at the proposal text and think "this would be a useful feature to have, I obviously would like it in the language". But because it's such an obvious feature to put in, it would be great if people ask themselves why the Go team didn't put it in in the first place. Because there are reasons and these reasons need answering.

This is exactly what I attempted to do here. I wrote this at 1am on the last legs of a cup of coffee, and it seems I failed to consider some scenarios in my comment. A simple "How would this address X and Y" would have accomplished the same effect without this lecture at the end.

[Merovius]

To be clear, this is what the proposal says about this question:

We could instantiate it at link time, but in the general case that requires the linker to traverse the complete call graph of the program to determine the set of types that might be passed to CheckIdentity. And even that traversal is not sufficient in the general case when type reflection gets involved, as reflection might look up methods based on strings input by the user. So in general instantiating parameterized methods in the linker might require instantiating every parameterized method for every possible type argument, which seems untenable.

Or, we could instantiate it at run time. In general this means using some sort of JIT, or compiling the code to use some sort of reflection based approach. Either approach would be very complex to implement, and would be surprisingly slow at run time.

Or, we could decide that parameterized methods do not, in fact, implement interfaces, but then it's much less clear why we need methods at all. If we disregard interfaces, any parameterized method can be implemented as a parameterized function.

So while parameterized methods seem clearly useful at first glance, we would have to decide what they mean and how to implement that.

The solution you suggest seem a variation on the second option here.

[mccolljr]

The solution you suggest seem a variation on the second option here.

I suggested a couple of different things, but yes, one of those suggestions would appear similar to the second half of the second option.

However, a generic function/method with interface constraints is quite similar to a version that simply accepts/returns a value of that interface type. In such a case the fallback would be a version that uses an implementation whose signature simply replaces uses of the parameter type with the interfaces types used in constraints. If the constraints are a type set, then an implementation using a type switch could be generated to account for the possible types. Of course, if the proposal to allow use of type sets in the same positions that normal interface types can be used is accepted, then this approach could capitalize on that feature, as well.

Of course, an obvious problem with substituting interface types is that the usages like var zero T have different meanings than when the function/method is instantiated with a concrete type argument. Perhaps, then, it would be necessary to have a "generic context" attached to a generic method to provide things like __runtime_generic_zero(reflect.Type) interface{}.

Are these things easy? Certainly not. I'm not suggesting they are. If anything, there are multiple proposals that would need to be accepted before generic methods could be considered approachable.

This proposal is on hold right now. That means that there is time to discuss possible implementations, weigh the pros and cons of each, discuss complexity and ways to mitigate it, etc, etc. It is possible that I am misreading your tone as well, but these comments seem a bit combative given that I am not at all saying "this is how it should be done". I have indeed re-read the proposal regarding methods since your original post, and I do not think that I am rehashing the points there. I think that similarity is not the same as the same.

So while parameterized methods seem clearly useful at first glance, we would have to decide what they mean and how to implement that.

This is the discussion I am trying to jump-start with these comments - what does a generic method mean? How can it be used? What limitations should/could they have? Your examples above were helpful in expanding that discussion. Perhaps you disagree with the suggested approaches. Is there a different approach that you think would work? Would your assertion be that these are too complex to add to the language at this time? Those are all valuable pieces of input that it seems like you'd be able to provide.

[smoyer64]

I think that the example issue can be approached the same way as Java does: using interface{} behind the scenes and panic if the customer did a bad assignment

This is a bad idea in my opinion - Type erasure is one of the most annoying limitations of generics in Java. I think it would go against the grain of the simplicity that Go aims for.

So true ... as a long-time Java developer we were waiting for reification from the time generics were introduced. Reification is a much harder problem than prohibiting type erasure and as I've aged I've come to like my languages strongly-typed.

[jhbrown94]

This comment is a narrow response to the no-parameterized-methods issue, and specifically this bullet: "we could decide that parameterized methods do not, in fact, implement interfaces, but then it's much less clear why we need methods at all."

I lack the Go expertise to speak to the implementation challenges -- especially as regards reflection -- , but parameterized methods on concrete types alone would solve a practical syntactic problem in that they would let you create type-safe pipelines without temporary variables or deep nesting. Type-safe pipelines reduce the need to rely on interfaces, so it'd probably be a fine tradeoff to limit such methods to concrete types.

So with current generics, to perform multiple type transforms without deep nesting, I have to use multiple temporary variables. Say I have a Box type, and a generic Map function:

	b := Box{5}
	b1 := Map(b, strconv.Itoa)
	b2 := Map(b1, strings.NewReader)
	b3 := Map(b2, moreMapping)
	return b3

We can use nesting to get rid of the temporaries, but that's worse:

	b :=
		Map(
			Map(
				Map(Box{5},
					strconv.Itoa),
				strings.NewReader)),
		moreMap)

But with Map as a parameterized method on the concrete Box type, we could do this:

	b := Box{5}.
		Map(strconv.Itoa).
		Map(strings.NewReader).
		Map(b2, moreMapping)

Again, I can't speak to the impact on reflection and other implementation challenges, but I would find this extremely helpful in taking advantage of generics to make more code fully typechecked at compile-time.

Quick edit to add: a day into playing with them, I am loving the generics already. Many thanks to everyone involved in their arrival!!

[asv7c2]

Just disallow type assertion from interface which contains methods with type parameters and disallow type assertion from empty interface to struct/type which contains methods with type parameters.

[mccolljr]

and disallow type assertion from empty interface to struct/type which contains methods with type parameters.

Can you explain this part further?

It seems to me that this would not require knowing anything about the generic methods, just the identity of the type. After that, the compiler knows the type and any usages of generic methods would be done from a known concrete type for which the compiler could generate implementations.

[asv7c2]

@mccolljr

You right, compiler know the type after type assert, I missed that.

[wxblue]

type A struct {
	Id  int
}
slice := []*A{
	{Id: 1},
	{Id: 2},
	{Id: 3},
}

type Selector[TIn any, TOut any] func(TIn) TOut
func Select[T any, TOut any](q Query[T], selector Selector[T, TOut]) Query[TOut]

b := Select(FromSlice(slice), func(p *A) int {
	return p.Id
}).Where(func(p int) bool { return p > 1 })

This works fine, but breaks the chained call. Maybe add some syntactic sugar like extension methods in C#.

func Select[T any, TOut any](this q Query[T], selector Selector[T, TOut]) Query[TOut]
b := FromSlice(slice).Select(func(p *A) int {
	return p.Id
}).Where(func(p int) bool { return p > 1 })

[seborama]

In the context of the provided example.

Isn't:

func (S) Identity[T any](v T) T { return v }

syntactic sugar for:

func Identity[T any](_ S, v T) T { return v }

?

While p4 is responsible for instantiating p1.S, p3 only is responsible for instantiating Identity[int].

I feel there is a dichotomy in the instantiations which perhaps helps resolve this conundrum?

Apologies, this may be an inadequate simplification of the issue. I don't have the knowledge of compiler theory and much less how the go compiler works.

I do feel that without finding a suitable solution for this issue, we may never see functional patterns in Go that include mapping from the receiver type to another parameterised type.

E.g.:

```go
func (mt MyType[T]) Map[R any](p1 Function[T,R]) MyType[R] { ... } // DOES NOT COMPILE
```
This can be achieved with:
type ReType[T any, U any] struct {
	from MyType[T]
	to   MyType[U]
}

func (ds ReType[T, R]) Map(mapper Function[T, R]) MyType[R] { ... }

But the syntax is awkward because chaining takes place right-to-left instead of left-to-right, that is:
ReType(ReType(my1, m2).Map(), m2).Map()

vs
m1.Map().Map()

[ianlancetaylor]

@seborama Identity is only defined in package p1. Package p3 does not import package p1. There is no way for p3 to know that it needs to instantiate a function in a different, unrelated, package.

[seborama]

Does that mean we are reaching the limits of what can be achieved with implicit interfaces? What would the introduction of explicit interfaces do to help with this issue and to the language? (sorry, I will refrain from posting further - my passion is taking me far out of my depth)

[ianlancetaylor]

As far as I can see explicit interfaces bound the problem but they do not solve it. The key to the example is that p3 needs to know about p1, but it only knows about p2. Explicit interfaces would mean that anything that imports p4 would be given additional information about p1, but p3 doesn't import p4.

[frederic-gendebien]

According to the Type parameters proposal, it is not allowed to define type parameters in methods.

This limitation prevents to define functional-like stream processing primitives, e.g.:

func (si *stream[IN]) Map[OUT any](f func(IN) OUT) stream[OUT]

While I agree that these functional streams might be unefficient and Go is not designed to cover this kind of use cases, I would like to emphasize that Go adoption in stream processing pipelines (e.g. Kafka) is a fact. Allowing type parameters in methods would allow constructing DSLs that would greatly simplify some existing use cases.

Other potential use cases that would benefit from type paremeters in methods:

• DSLs for testing: Assert(actual).ToBe(expected)
• DSLs for mocking: On(obj.Sum).WithArgs(7, 8).ThenReturn(15)

I also tried to implement Streams and Functional Pipelines, just stopped because type parameters don't allow this... This is just insane! Either we do it right, or we don't it at all...

[mariomac]

(off-topic for @frederic-gendebien, feel free to remove the comment if you think it shouldn't be here)

Despite I am the first wanting type parameters in records, the current implementation is a big step forward and I think it's great. What I did to overcome this limitation in my streams library:

• Define methods for these cases where the input and output types are the same.
• Define functions for the cases where the input and output types differ. You can have a look at this example: https://github.com/mariomac/gostream#example-3-generation-from-an-iterator-map-to-a-different-type

[seborama]

Is there such a concept as a compile-time call-graph?

From p3 outwards, there is no way to tell how to instantiate vi.Identity[int](0) because p3 doesn't know what v is by itself.

When a parameterised method is referenced by interface (with the parameter type not provided by the interface), could the compiler defer the analysis to where p3.Identity is actually being called (i.e. p4 in this example)? By proceeding inwards, the compiler would be able to follow the call-graph from p4 (with contextual knowledge of p1.S{}) all the way down to S1.Identity in p3.

I have probably overlooked something.

[Merovius]

@seborama Consider that the interface value might have also been sent over a channel or put in a map or a slice with a lot of other dynamic types and shuffled around a bit. There is simply no way, in general, for a compiler to conclusively say what dynamic types could be in a given interface. It can (and will) do that analysis in some cases, e.g. to devirtualize interface-calls for performance. But this can never be more than a heuristic and can't affect the correctness of a Go program.

[ianlancetaylor]

@bep The problem is that when the code is split across several packages there is no way to instantiate the method. See https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods.

[changkun]

Is explicit instantiation a possible option here? We already have underscore _ import for packages, also for checking a struct to implement an interface. What might be a problem for the following:

We have a struct X that implements an interface I:

type I[T any] interface {
	Foo(T)
}

var _ I[int] = &X[int]{} // OK

type X[T any] struct {
	t T
}

func (x X[T]) Foo(v T) {}

Let's assume that we expect to extend X with a method Bar that has additional type parameters, such as U:

func (x X[T]) Bar[U any](u U) {}

It looks like we are implicitly expanding the type set [T any, X[T]] to satisfy the following type set:

type J[T any, U any] interface {
	I[T]
	Bar(U)
}

Hence, when X includes method Bar, this type check might expected to fail, and throw message like this:

var _ I[int] = &X[int]{} // ERROR: X does not implement interface I (X has method Bar with type parameters)

Instead, we could manually and explicitly resolve the instatiation by:

var _ J[int, int] = &X[int]{} // OK: and instantiates (X[int]{}).Bar[int]

With this approach, whenever we have code like this:

func check(x any) {
	if _, ok := x.(interface { Bar(int) }); ok {
		// ...
	}
}

The runtime assertion will only return false unless we have the mentioned explicit instantiation.

---

For the mentioned example in the proposal, we could do:

package p1

// Instantiates method explicitly
var _ p2.HasIdentity[int] = &S{}

type S struct{}

func (S) Identity[T any](v T) T { return v }

package p2

type HasIdentity[T any] interface {
	Identity(T) T
}

---

One clear downside of this approach: it is required us to explicitly states all method instantiations. However, it is worth understanding how often this runtime assersion-related instantiation problem will occur.

[Merovius]

@changkun That would mean that a harmless change in one package can break the behavior of another. Say, package a is

type S struct {}
func (S) F[T any](T) {}
type HasF[T any] interface { F(T) }

package b is

var _ a.HasF[int] = a.S{}
func G() {
    var x a.S
    x.F(42)
}

and package c is

type HasF interface { F(int) }
func G(v interface{}) { fmt.Println(v.(HasF).F()) }

Now, the author of package b decides that it doesn't actually need the F[int] method on S after all and removes the _-assignment. This seems a benign change - after all, a.S is not part of any API and this change is neither. But now, c.G panics when called with an a.S.

Fundamentally, this is the same problem why we disallow foreign packages adding methods to types - we don't want interface-satisfaction to change depending on what packages are imported in a program. So, we would have to use the same solution, which is that only the package containing the type could do those "method instantiations". But that severely limits the use of type-parameters in methods. For example, you still could write chainable methods on a custom container type, but all instantiations would have to be done by the container, so wouldn't work with user-types - so why have type-parameters on them at all?

There's also the general usability aspect. The point of adding type-parameters to methods is to avoid having to write somepkg.F(somepkg.T{}) in favor of somepkg.T{}.F(). But if you have to explicitly instantiate the methods before using them, most of that usability-benefit vaporizes - I don't think

type X interface[T any] { F(T) }
var _ X[myType] = somepkg.T{}
func G(v somepkg.T) {
    var w myType
    v.F(w)
}

is in any way better than writing

func G(v somepkg.T) {
    var w myType
    somepkg.F(v, w)
}

Lastly, I personally just don't like the idea of _-assignments having these sort of side-effects. It seems very magical. But that's really the least of its problems.

[changkun]

a harmless change in one package can break the behavior of another

This is an interesting view, and I guess it might also depend on people's usage. If one uses underscore for explicit instantiation, potentially it should come with a comment to state the purpose there. Hence removing it might also very likely be a change on purpose (?) and expecting to break something.

this is the same problem why we disallow foreign packages adding methods to types - we don't want interface-satisfaction to change depending on what packages are imported in a program.

This might permit additional instantiation from foreign packages, but if a package author writes a parametric method, they should anticipate the package to work for any instantiated types. Under this context, should it be considered harmful? It looks like a similar effect as we already had with init() or compiler magic, such as import _ "image/jpeg" and import _ "embed" or so, although it had some debate in the past.

functionally this isn't actually very different from just doing [...] and instantiating X[int, int] instead of X[int]

I'd argue this is not something we wanted because whenever we try to extend the method of X, we will have to change its type parameters. Then, all places that use X must be altered and required to fill more type parameters. The main difference is that explicit instantiation does not require changing the existing parameter list of X.

For instance,

type X[T any] struct { … }
func Foo[T any](x X[T]) { … }

When extend X with a new parametric method F:

-type X[T any] struct { … }
+type X[T, U any] struct { … } // U may also not appear in any of the struct fields
+func (X[T, U]) F(U) { … }

And we also have to change all possible references to X[T] (including call side)

func Foo[T, U any](x X[T, U]) { … } // U may have no actual functionality in the body

I think we could agree that this above situation must be avoided if we found a way to permit type parameters in methods.

[Merovius]

If one uses underscore for explicit instantiation, potentially it should come with a comment to state the purpose there. Hence removing it might also very likely be a change on purpose (?) and expecting to break something.

The problem is that using this mechanism is an implementation detail, but it now becomes API surface. The main intended use-case for this is functional chaining for custom container types. You wouldn't expect the fact that you are using that in your code to suddenly become part of your API.

Again, the change I outlined above appears completely benign. No one has any way of even knowing package b imports package a, without looking at the code. It is completely unexpected for something like that to be part of the API of a package.

It looks like a similar effect as we already had with init() or compiler magic, such as import _ "image/jpeg" and import _ "embed" or so, although it had some debate in the past.

embed is misplaced here, as it can't break behavior, but sure. import _ "image/jpeg" and the like can absolutely break programs. But a) that's not a good argument to add more of a bad thing and b) that's why these imports are generally recommended to only appear in package main. So that these breakages don't happen. That's different here - using the method requires instantiating it, so it usually happens in library code.

functionally this isn't actually very different from just doing [...] and instantiating X[int, int] instead of X[int]

Sorry, I retracted that comment by deleting it. I was waving my hands around too vigorously and it was beside the point anyways.

[AaronFriel]

@ianlancetaylor to add another data point for allowing generic methods on structs until someone is able to solve the same for interfaces, github.com/pulumi packages contain at least 330,000 implementations of an interface Output, which is a Promise-like interface designed to carry with it some metadata that accumulates. This is an ideal use case for a concrete generic struct.

But the lack of generic methods makes the already unergonomic "ApplyT" method - a method which takes a function any -> any and returns an Output, i.e.: it's an untyped map operation - very difficult to read by putting it in prefix position and only accessible via package name. Expanding on the comment here https://github.com/golang/go/issues/49085#issuecomment-995993517 by @jhbrown94, this is how the pipeline might look:

b := // let this be an Output[int]
b1 := pulumi.Apply(b, strconv.Iota) // Output[string]
b2 := pulumi.Apply(b1, strings.NewReader) // Output[*strings.Reader]
...

Or alternatively:

b := // Output[int]
x := pulumi.Apply(pulumi.Apply(b, strconv.Itoa), strings.NewReader)

If one of those functions was instead a function literal it would be fairly unreadable.

Compare this with a method based approach, which isn't unreadable even with expanded functions. Arguably, it's more readable by allowing the types to be explicit in each application:

b := // Output[int]
x := b.Apply(func (x int) string {
  return strconv.Itoa(x)
}).Apply(func (x string) *strings.Reader {
  return strings.NewReader(x)
})

This expresses the computation in a linear, readable order.

[venth]

I think that usage of method reference instead of declaration of an anonymous function, will increase readability. Although I think that would be better to describe it in a separate proposal.

[jpap]

I am most interested in this proposal because I'd like to use chained generic method calls like the OP. It improves the DX for projects like RxGo and I face similar challenges in my own work because of this. @mariomac's workaround is pragmatic, but still suffers from the same degraded DX.

I don't necessarily have a need for using such types with interfaces, but I understand that is the main blocker here.

Addressing the "No Parameterized Methods" example

I propose two constraints:

1. Generic interfaces must be made "concrete" when used in a type assertion, within a type switch, and when invoking associated methods. By concrete, I mean that all type parameters on the generic interface are specified.

2. Types having generic method(s) cannot be typecast to interface{} or a generic interface unless it is an "instantiated generic type". By instantiated generic type, I mean a type that has one or more generic method instantiations.

Constraint 1 means that a type assertion must be conditioned on the parameter types, and therefore would only be valid if suitable method instantiation(s) exist. When the assertion is valid, we can go ahead and invoke the method as if it were no-longer generic and know that it will not panic.

Constraint 2 forces us to explicitly instantiate generic methods at compile time, without having to resort to the linker workaround described in the above example endnote. (This instantiation will typically occur near the site where the generic type is created; cf. other proposals that "pre-declare instantiations" OOB near the generic type definition.) To do this, we first typecast such a generic type to a concrete interface, which specifies the type parameter set(s) of that interface's method(s). This triggers the compiler to generate all related method instantiations at compile-time. We now have an concrete interface type that can be typecast to interface{} or be assigned to a compatible generic interface that can be used with Constraint 1.

I hope this makes more sense after reading the sample code below. Please expand to read.

Proposed Go syntax (sample code)

package p1

// S is a type with a parameterized method Identity.
type S struct{}

// Identity is a simple identity method that works for any type.
func (S) Identity[T any](v T) T { return v }

package p2

// HasIdentity is an interface that matches any type with a parameterized
// Identity method.
//
// In this proposed syntax, all method type parameters are declared with the
// interface.  Each method can use a subset of the list of type parameters
// declared, and is similar to a generic Swift protocol declaring a set of
// "associated types"; see the Mapper type for an example having multiple
// types.
//
// This generic interface can then be made concrete by specifying all of the
// types.  e.g. HasIdentity[int] is a fully specified concrete interface.
type HasIdentity interface[T any] {
	Identity(T) T
}

// Mapper illustrates how multiple type parameters are declared on the
// interface.  (This interface is not used by the sample code.)
//
// A corresponding concrete interface: Mapper[int, string]
type Mapper interface[T, OutT any] {
	HasIdentity[T]
	Map(func(T) OutT) []OutT
}

package p3

import "p2"

// CheckIdentity checks the Identity method if it exists.
func CheckIdentity(v interface{}) {
	if vi, ok := v.(p2.HasIdentity); ok { // INVALID: interface must be concrete
		// ...
	}
	if vi, ok := v.(p2.HasIdentity[int]); ok { // OK: concrete interface
		// Identity is invoked below as if it were no-longer generic because the
		// type parameter is fixed by the type assertion above.  This assertion is
		// assumed to be true when v is an instantiated generic type over the int
		// type parameter, where the program definitely has an associated method
		// implementation.
		if got := vi.Identity(0); got != 0 {
			panic(got)
		}
	}
}

// CheckIdentityTyped is a type-specific version of CheckIdentity
func CheckIdentityTyped(v p2.HasIdentity) {
	v.Identity[int](0) // INVALID: interface is not concrete
	v.Identity(0)      // INVALID: as above, with inferred type parameter

	if vi, ok := v.(p2.HasIdentity[int]); ok { // OK: concrete interface
		if got := vi.Identity(0); got != 0 {
			panic(got)
		}
	}

	// The following is OK, but could panic before we call the Identity method
	// if v's underlying type does not have generic methods instantiated over
	// int at compile-time.  Such a panic is conceptually similar to a failed
	// type assertion on a non-generic type.
	if v.(p2.HasIdentity[int]).Identity(0); got != 0 {
		panic(got)
	}
}

package p4

import (
	"p1"
	"p2"
	"p3"
)

// Define concrete interfaces over the uint and int types.
//
// We do this over two types for illustrative purposes: in practice, you'd
// instantiate only the types that you need.
//
// The linker might be able to drop instantiations that remain unused, where
// as per usual the reflect package is not in use, because we know that calls
// to methods must be made via concrete interfaces that explicitly specify the
// type parameters and these can be checked against all those instantiated.
type Integer interface { uint | int }                // no ~ operator here!
type HasIntegerIdentity p2.HasIdentity[Integer]      // concrete interface
type HasIntegerIdentityEq p2.HasIdentity[uint | int] // inline equivalent to previous line

// CheckSIdentity passes an S value to CheckIdentity as interface{}.
func CheckSIdentity() {
	p3.CheckIdentity(p1.S{})                                  // INVALID: type with generic method cannot be directly typecast to interface{}
	p3.CheckIdentity(p2.HasIdentity[uint|int](p1.S{}))        // instantiate p1.S.Identity[uint], p1.S.Identity[int]
	p3.CheckIdentity(HasIntegerIdentity(p1.S{}))              // equivalent to previous line
	p3.CheckIdentity(HasIntegerIdentityEq(p1.S{}))            // equivalent to previous line
}

// CheckSIdentityTyped passes an S value to CheckIdentityTyped as a typed interface.
func CheckSIdentityTyped() {
	var x p2.HasIdentity = p1.S{}                             // INVALID: type with generic method cannot be directly typecast to generic interface
	var y p2.HasIdentity = p2.HasIdentity[int|string](p1.S{}) // instantiate p1.S.Identity[int], p1.S.Identity[string]
	p3.CheckIdentityTyped(y)
}

On the reflect package, I propose we simply add a new "law of reflection" that says that any compile-time uninstantiated type realizations are unavailable at run-time.

Here's how that document currently describes reflection, my emphasis added:

Reflection in computing is the ability of a program to examine its own structure, particularly through types; it’s a form of metaprogramming. It’s also a great source of confusion.

In the same way that the "reflect" package doesn't allow us to create a new func with arbitrary code body, add a new method to an existing struct, or create a new arbitrary type, I think it is reasonable not to expect "reflect" to create instantiations of generic methods that are not already part of the program structure. That is, I think we should allow reflect to say "such a generic method instantiation is unavailable", rather than create it at runtime, JIT or otherwise.

The "reflect" package could be extended to query for this, conceptually similar to how CanAddr reports whether a value is addressable.

---

I hope there are no holes here, but if there are I hope we can find a path with suitable constraints on generic types and interfaces so that we can ultimately enable the OP chaining scenario. This would make Go generics even more awesome. 🎉

[Merovius]

@jpap

I have to extrapolate a bit, because you use uncommon nomenclature and leave a couple of details implicit. So, to summarize my understanding of what you suggest:

1. Generic interface types (that is "interface types with a type parameter") should be usable as types without requiring instantiation. E.g. you can have var v HasIdentity, with type HasIdentity[T any] interface{ Identity(T) T }.
2. It is invalid to call methods on such types. To call any method, you first need to type-assert them to a non-generic interface type. E.g. you have to type-assert w := v.(HasIdentity[int]) so you can call w.Identity(w)
3. It is allowed to add extra type parameters to method declarations. Let's call this a "generic method". That is, it's allowed to write func (S) Identity[T any](v T) T { return v }.
4. A value with a generic method can not be assigned to any or a generic interface type (question: Why single out any here? I can also type-assert an io.Reader to have an Identity method. I don't understand what makes any special here). That is, it is not allowed to write HasIdentity(v) or any(v) (and question: Is it allowed to write io.Reader(v), assuming v also has a Read method with the right signature? If so, why is this allowed but not any?)
5. A value with a generic method can be converted to an interface type, as long as that interface type a) has a method with the same name which b) has the same signature as the generic method, when all type parameters in the latter are replaced with some concrete type. That is, it is allowed to write HasIdentity[int](v) or HasIdentity[string](v). We call such a conversion an "instantiation of the generic method".
6. An interface value can be assigned to a generic interface, as long as it is a valid instantiation of that interface. That is, it is valid to assign a HasIdentity[int] to a HasIdentity.
7. If an interface value's dynamic type has a generic method, a type-assertion to an interface with that method (replaced with concrete types for its type parameters) only succeeds if the program contains an instantiation of that generic method to those type arguments. That is, v.(HasIdentity[int]) would succeed, but v.(HasIdentity[time.Duration]) would fail (if we did the method instantiations as above).

Is this a reasonable description? If so

1. You don't address what I brought up here. That is, you talk about how conversions to and from interfaces work. But even without interfaces, generic methods create problems, as they de-facto allow to pass around uninstantiated generic functions. We don't allow that currently, because it puts unacceptable constraints on the implementation strategy.
2. Most of the apparatus you introduce doesn't seem to really have a point. If a function has to type-assert a generic interface down to its concrete instantiation to call the method - why even accept a generic interface? Why would you write func CheckIdentityTyped(v p2.HasIdentity) if you could also just write func CheckIdentityTyped(v p2.HasIdentity[int]) if that's the instantiation you are ultimately interested in?
3. It seems incredibly cumbersome to have to type-assert every method call. A lot of the use-cases for generic methods come down to chaining them. This wouldn't be possible, as every method call would have to be interceded with type-assertions. I don't think this is satisfying in actual usage.
4. The problem I mention here also exists for your suggestion. Seemingly harmless, backwards compatible changes in a package can now cause any program including it to crash. With no ability to detect it. That's because you still suggest the same semantics for interface type-assertions, even if you use a different syntax for "instantiations of a method".

At least if I understand you correctly, I don't think this solves any of the real challenges we have with generic methods, while also being pretty cumbersome and confusing to use in practice. So personally, I don't think this is a good design.

[mccolljr]

@Merovius

It seems like the 2 largest problem areas you focus on are:

1. Generic methods during reflection
2. Generic methods on interfaces

I'm not sure I understand why it would be infeasible to just... not support these things.

A Generic method on an interface value implies that the runtime will always find or generate the requested implementation. Since we don't want/need runtime code generation and since we do not require (or even have) explicit implementation, I don't see any way this could feasibly be supported in the language as it is today. It seems like a very straightforward decision to disallow this entirely.

As for reflect? Programs that rely on reflect (and interface{} returns/arguments, for that matter) are already susceptible to surprise breaks in downstream package changes. Current users of reflect already need heavy testing for reflect-based code.

Blocking this extremely useful feature because we can't figure out a good way to make it work with reflect seems like a perfect example of "making perfect the enemy of good".

Reflect already has limitations around genetics. For example, if we have type Blah[T any] interface {}, there is no way to, at runtime, ask whether a reflect.Value implements Blah[T] for some dynamic T known only at runtime... which seems normal and expected to me, since generics are a compile time feature.

As such, reflection doesn't have a responsibility to support generic methods. Perhaps they can be supported as far as appearing in the method list, and perhaps their entry in the method list can refer to a vtable to query whether it is implemented for a specific concrete type, and perhaps if an implementation is found it can even be callable. But I see no reason for reflect to require an ability to call a generic method at runtime for a type whose instantiation was not compiled.

There is also the question about how a concrete type with generic methods implements an interface. For example, given:

type Blah struct {}

func (b Blah) [T any] Identity(v T) {
    return v
}

type Identifier[T any] interface{
    Identity(v T) T
}

...we need to decide whether Blah implements Identifier, and if so, how we can check that at runtime.

And with this... my argument is the same as it is for reflect: code using runtime type assertions and type checks already needs to be tested, and is already susceptible to breaking changes in downstream packages.

As such, it would be reasonable to say that assigning a value of concrete type Blah (or *Blah) to a variable or struct member of type Identifier[T] for some T at compile time would be valid, and the compiler could generate code for that instantiation of Blah.Identity, but the same check at runtime could fail if Blah doesn't have a compiled implementation for the necessary type T.

In the wider world, how many people are going to be using reflect or interface assertions with types using generic methods? It is my understanding that to some extent the generics feature is meant to alleviate some of the need for this type of "unsafe" code.

Generic methods on concrete types would go a long way to drive adoption of the generics feature in the wider ecosystem. Many libraries would benefit from it, and it would complement other Go features such as channels quite well. Overall, it seems to be a quite popular proposal. As such, deciding on a set of practical limitations to get this feature into the discussion in a serious way seems like it should be a priority.

I said it before, and I'll say it again: you clearly have a lot of knowledge and context around this feature and its implementation details. Your asserions that other people's proposals are incomplete necessarily imply that you have some idea what pieces are missing. You could always choose to use that insight to drive the conversation forward by making suggestions, rather than simply dropping by this issue to level criticism.

[atdiar]

Hmmh, maybe if we step back a little, the problem can be reframed because I am not sure that non-parametered types should even support parametered methods.

Rather, I feel that the type parameters of the methods should belong to a generic type definition.

Also, because there is no variance of type constructors, generic methods can at best suggest implementation of interfaces lacking similarly-named methods(unless the type set of the method's parameter constraint is a singleton) or a generic interface.

The issue still is that because method names are not parametric, generic methods may introduce conflicting names depending on the type argument (only one method signature is accepted per method name). This is something to take into account when evaluating whether a type with generic methods satisfy a constraint.

It's quite unlikely for any generic method to suggest implementation of a specific basic interface with current Go.

At least, we can't rely on it in generic code the same way we can't rely on any exact same constraint that is not shared by every member of a type set.

Just food for thought.

[Merovius]

@mccolljr

It seems like the 2 largest problem areas you focus on are:

No, personally I think the largest problem is the one I mention here. My last comment focuses on interfaces, because that's what the suggestion I respond to did.

[Merovius]

@mccolljr

I said it before, and I'll say it again: you clearly have a lot of knowledge and context around this feature and its implementation details. Your asserions that other people's proposals are incomplete necessarily imply that you have some idea what pieces are missing. You could always choose to use that insight to drive the conversation forward by making suggestions, rather than simply dropping by this issue to level criticism.

I am pointing out what pieces are missing. I don't have suggestions, because I believe they are missing. That's the "insight" I am offering (though really, I don't have special knowledge or context. I'm literally just re-hashing the design doc): Without some radically fresh ideas, allowing parameterized methods would make the language inconsistent and/or violate some design criteria for generics in Go (e.g. not requiring a specific implementation strategy). I don't think there is anything wrong with defending that position. And I don't believe it is reasonable to put the responsibility for having those radically fresh ideas on other people.

[Merovius]

FWIW here's another way to look at it, which IMO speaks in favor of talking about passing around generic functions/types before tackling generic methods.

On the implementation level, an interface is just a struct with a bunch of func fields (the itab) and compiler automatisms to extract those fields and populate them when assigning to an interface type. It's paired with a reflect.Type, to be able to do type-assertions (in the case of an interface type-assertion, this means looking up methods by name to populate the itab). For example, io.Reader is conceptually a struct{ Read func([]byte) (int, error) } and when assigning an *os.File to it, the Read field is set to (*os.File).Read.

When we think about how this would work with generic methods, the most natural way would be to represent e.g. HasIdentity from the design doc as a struct{ Identity func[T any](T) T }. But doing that obviously first requires us to represent func[T any](T) T - that is, we need uninstantiated generic functions.

In fact, basically every question about interfaces in the presence of generic methods is naturally answered, if we where able to use uninstantiated generic functions as values with normal types.

A type-assertion to a generic interface is hard in exactly the same way that a type-assertion to an uninstantiated generic function is hard. That is, if we assume we can pass around generic functions uninstantiated, we can can replace HasIdentity in the example with type Identity func[T any](T) T and we'd run into exactly the same problems, no methods or interfaces required:

package p1

// Identity is a simple identity function that works for any type.
func Identity[T any](v T) T { return v }

package p2

func CheckIdentity(v any) {
	if f, ok := v.(func[T any](T) T); ok {
		if got := f[int](0); got != 0 {
			panic(got)
		}
	}
}

package p3

import (
	"p1"
	"p2"
)

func CheckSIdentity() {
	p3.CheckIdentity(p1.Identity)
}

If func[T any](T) T was a valid type, it would be pretty obvious how interface satisfaction and type-assertion would work:

1. func (S) Identity[T any](T) T would make S implement type HasIdentity interface{ Identity[T any](T) T }
2. It would not make it implement type HasIntIdentity interface{ Identity(int) int }, as the Identity method of S has type func[T any](T) T, which would be a distinct type from func(int) int. This is very easy to explain.

Interface type-assertions could just look up the method by name and it would be a representable (though uninstantiated) func value. All of this could work exactly as it does now - just that the func is generic.

Basically, all the infrastructure, all the compiler-magic and what reflect does - it's all already in place or very easy to nail down. Except the crucial question of how we'd represent an uninstantiated generic function and how we would instantiate and call it. So, not only does allowing generic methods already require us to figure out how to pass around uninstantiated generic functions - once we do, allowing type parameters on methods is almost a corollary. The two problems seems mostly equivalent to me. But at least, when we talk about passing around uninstantiated generic functions, we don't get bogged down in syntax or the abstract semantics of interfaces.

Of course, this leaves us at the unfortunate point in the discussion where we don't know how to do that, without violating the design criteria of generics in Go - most importantly, not to prescribe a particular implementation strategy. As far as I can tell, it's unlikely this will get resolved anytime soon. Because the options and tradeoffs seem to be mostly clear and well-explored. And the conclusion seems to be "we can't". So, moving that forward would require someone to come up with a real convincing case why one of the implementation strategies is obviously the right one. But, waiting for that argument is why Go didn't have generics for a decade in the first place.

And it's probably frustrating to see this feature as very useful (I agree) and being told that it's basically impossible to solve anytime soon. But, again, it took a decade to get generics because of these same issues. Sometimes there just isn't a good answer.

[atdiar]

That's what I thought too initially but then I edited to be more conservative. For instance, does HasIntIdentity implement HasIdentity?

This looks like introducing some kind of variance.

Also, S is still strangely defined. I believe that it would be preferable to define types with generic methods as such: type S[T constraint, M methodconstraint....] struct{}

Otherwise, we don't know what S implements (it might have an instantiated method with the correct type signature that still doesn't implement the related interface) and it looks like it can create some kind of spooky action at a distance. Parametered types are essentially needed to be type constructors. Note sure we need to deal with the general case of abstract types. Making a higher order concept interact with the rest of the code might add a bit too much complexity.

But I am still on the fence because this problem doesn't seem to be that much more different from what is done with type operators as well. (comparable for instance)

[Merovius]

For instance, does HasIntIdentity implement HasIdentity?

No. That would make no sense. However, the other way around it's a valid question to ask. But

1. I think the obvious way to do it is to start with "no" and then consider "yes" in the future
2. Either way, we could implement both options - if we know how to instantiate a generic function at runtime, HasIdentity could implement HasIntIdentity via an implicit instantiation if we think that's necessary and
3. There's no need to ask this question if we don't even know how HasIdentity can even exist.

[atdiar]

I may be mistaken but it makes more sense to me to consider whether HasIntIdentity implements HasIdentity than the opposite.

The same way each comparable type has a different version of the comparison operators but they all are said to implement the comparison interface.

Or the addition operator for interface{int | float...} which is implicitly tied to the type parameter somehow.

I think it needs much more thought.

[Merovius]

I may be mistaken but it makes more sense to me to consider whether HasIntIdentity implements HasIdentity than the opposite.

HasIntIdentity means you have a method Identity(int) int. HasIdentity means you have a method Identity[T any](T) T. So if HasIntIdentity would implement HasIdentity, you'd have to be able to make a func[T any](T) T if I give you a func(int) int. That makes absolutely no sense.

It makes a certain sense for HasIdentity to implement HasIntIdentity, OTOH. Because if the type has a method Identity(T) T for any type, it obviously could have a method Identity(int) int.

[atdiar]

Well I think we understand things a bit differently. HasIntIdentity implementing HasIdentity may mean that there is existence of a type argument for which HasIdentity is HasIntIdentity.

On the other hand, HasIdentity wouldn't implement HasIntIdentity lest the type parameter constraint has a type set that is the singleton {int} I guess. So in general, that wouldn't be the case.

Said colloquially, HasIdentity would represent the set of all types that have an identity method. HasIntIdentity's type set would be included in that set then.

Perhaps that I am wrong but already, this isn't very obvious to me.

[Merovius]

@atdiar There are three different interfaces we could consider

type HasIdentity1 interface { Identity[T any](T) T }
type HasIdentity2[T any] interface{ Identity(T) T }
type HasIntIdentity interface{ Identity(int) int }

You seem to be thinking of HasIdentity2 when you say HasIdentity. HasIdentity2 is already expressible today and yes, today HasIntIdentity implements HasIdentity[int]. In fact, the two are identical.

But in the scope of this issue, HasIdentity1 is the relevant one (as it is the one that has type parameters on its method and it's the one mentioned in the example in the design doc). And it makes categorically no sense for HasIntIdentity to implement that. Implementing an interface means you have all the methods. So for HasIntIdentity to implement HasIdentity1, any type which has a method Identity(int) int would need to be able to provide a method Identity(string) string - as that would be something you could get out of HasIdentity1. It's categorically non-sensical to have HasIntIdentity implement HasIdentity1.

[edit] It might make sense to ask if HasIntIdentity implements HasIdentity1 (the uninstantiated interface) which we could get if we allowed uninstantiated generic types. I'm not sure what it would mean to implement an uninstantiated interface. But, that's still a question we'd have to ask if we where to solve "passing around uninstantiated generic types", not when we try to solve "attaching type parameters to methods". So even then, that only re-inforces that we should talk about the former first [/edit]

[Merovius]

@atdiar If HasIntIdentity implements HasIdentity, this code would be presumably correct, so what would it do?

type HasIdentity interface{ Identity[T any](T) T }
type HasIntIdentity interface{ Identity(int) int }
func G(v HasIdentity) { fmt.Println(v.Identity[string]("foo")) } // v.Identity has type func[T any](T) T, so can be instantiated with string
func F(v HasIntIdentity) { G(v) } // HasIntIdentity implements HasIdentity, so is assignable to it 
type S struct{}
func (S) Identity(v int) int { return v/2 }
func main() { F(S{}) } // S implements HasIntIdentity, just based on normal interface rules

There just is no way for this to possibly work.

[atdiar]

Yes, I am not sure that HasIdentity1 should even be possible, the same way S as defined in the proposal seems strange to me.

I would consider allowing parametered methods on parametric types only. I guess I was thinking about HasIdentity2.

I might be missing something though. What would be the benefit of allowing parametered methods on instantiated/non-generic types?

[jpap]

Thanks for the read, @Merovius.

After preparing a response to your questions, which you can still read by expanding under Previous discussion below, I thought of a way to simplify it further as restated below. It takes the essential part of the last proposal, which is the explicit linkage between a type having parameterized methods and a generic interface.

Revised proposal

Types with parameterized method(s):

1. do not implement a generic interface unless explicitly adopted.

   The design doc describes a couple of undesirable options to make paramertized methods work in a natural way, then goes on to say (my emphasis added),

   Or, we could decide that parameterized methods do not, in fact, implement interfaces, but then it's much less clear why we need methods at all. If we disregard interfaces, any parameterized method can be implemented as a parameterized function.

   So while parameterized methods seem clearly useful at first glance, we would have to decide what they mean and how to implement that.

   Parameterized methods are relevant, because they allow for chaining of calls, which was the motiviation for the OP.

   If we default to the rule that parameterized methods do not implement an interface, we directly avoid the problem described in the design doc.

   By taking that rule as default, then allowing a type to opt back in explicitly for a given interface, we can also make parameterized methods for that interface cheaper to implement. (See below.)

2. can explicitly adopt one or more interfaces using a new predeclared function,

   func adopt(Type, ...Interface) Type
   

   which would require that Type can satisfy all of the generic Interfaces listed at compile time. i.e. it is a compile error if Type cannot possibly implement each Interface, given all of the type parameter constraints.

   At compile time, we use this as a decorator to determine how to instantiate generic methods. (See below.)

   At run time, this function would mark the adopted interface(s) against the type instance value. Subsequent type assertions for those interface(s) would be allowed specifically for this instance value. This avoids "spooky action at a distance", where changes at this call site inadvertently affect other unrelated parts of the program.

   How these marks are recorded is an implementation detail that needs to be worked out, but it is essentially a mapping of instance values to a list of opt-in interfaces.

On support for "reflect", I wish to import my previous comments.

Compile time method instantiation

To instantiate generic methods at compile time:

1. We can easily scan the AST for interface adoptions (adopt call sites). This gives us a mapping from types having parameterized methods to generic interfaces.

2. We can also scan the AST for call sites of generic interface methods. This gives us a list of the required type parameters for each invocation of a given generic interface.

We then combine results from each scan to determine method instantiations. They are not prescribed to any particular implementation strategy: they can be boxed, specialized, or whatever the current approach is for regular generic functions.

These scans can be performed together, separately, or in conjuction with any existing AST traversals during a build where type information is available. This is far simpler than analyzing the call graph.

Uninstantiated generic functions (UGF)

The following is with reference to @Merovius' more recent post and code example.

We could treat func Identity[T any](v T) T as a typed object in its own right, that can have zero or more implementations, each that is specialized to a given T. Its internal representation would include a mapping between a type-set, specifying a particular parameterization, and its implementation, which is just a pointer to the executable text.

When the type parameterization is fixed at a given call site, it is straightforward to instantiate the call and "bake it in": this is what a generics call site looks like today, e.g. for Identity[int](0).

In general, if we want to call a UGF via a type assertion, we need to do a lookup of the implementation at run time. This is the case in your CheckIdentity function, where the type assertion is against the UGF type, and assigns the object to f. The syntax f[int](0) selects the internal pointer implementation[int] at run time and passes execution to it with the given argument.

Even though we may require a run time lookup, it would be better if all of the implementations of a UGF can be determined at compile time so we avoid the need for a run time interpreter or JIT code generation. The question is then how to know what all the instantiations need to be. i.e. How do we find T = int in the previous example, especially when crossing two package boundaries?

One option is described in the design doc:

We could instantiate it at link time, but in the general case that requires the linker to traverse the complete call graph of the program to determine the set of types that might be passed to CheckIdentity.

Another cheaper option is scan the AST with type information for all type-assertions of func[T any](T) T and note the type parameters used at each subsequent call site. We could then create implementations for each type and bake them into the executable. This is more conservative, in that we might generate implementations for types that are not actually used with the Identity UGF, because some of the above call sites may never involve this particular UGF.

This process is quite similar to what I am proposing above for parameterized methods and generic interfaces, except that the idea of explicit adoption is redundant here because the "interface" being adopted is really just the UGF itself. In this case, we'd want to be opt-in by default (otherwise the UGF doesn't add much value). Since the equivalent "interface" is just the type itself, we also don't need to explicitly specify it.

I'm not proposing we implement UGFs: my interest is still with types having parameterized methods. But from this point of view, they are more like a special case of the generic interface problem, even without consideration of uninstantiated generic methods.

Sample code

The changes required to the sample code from the design doc are minimal, and are contained to `package p4` only.

package p1

// S is a type with a parameterized method Identity.
type S struct{}

// Identity is a simple identity method that works for any type.
func (S) Identity[T any](v T) T { return v }

package p2

// HasIdentity is an interface that matches any type with a
// parameterized Identity method.
type HasIdentity interface {
	Identity[T any](T) T
}

package p3

import "p2"

// CheckIdentity checks the Identity method if it exists.
// Note that although this function calls a parameterized method,
// this function is not itself parameterized.
func CheckIdentity(v interface{}) {
	if vi, ok := v.(p2.HasIdentity); ok {
		if got := vi.Identity[int](0); got != 0 {
			panic(got)
		}
	}
}

package p4

import (
	"p1"
	"p2"
	"p3"
)

// CheckSIdentity passes an S value to CheckIdentity.
func CheckSIdentity() {
	p3.CheckIdentity(adopt(p1.S{}, p2.HasIdentity)) // p1.S{} explicitly adopts interface p2.HasIdentity
}

func CheckSIdentity1() {
	x := p1.S{}
	x = adopt(x, p2.HasIdentity)
	p3.CheckIdentity(x) 
}

func CheckSIdentity2() {
	x := p1.S{}
	y := any(x)                  // OK: non-generic interface (backwards compatibility)
	y = adopt(y, p2.HasIdentity) // INVALID: y (of type any) does not implement p2.HasIdentity p3.CheckIdentity(y) by default
}

func CheckSIdentity3() {
	x := p1.S{}
	p3.CheckIdentity(x) // no panic: type-assertion is false, because x does not implement p2.HasIdentity by default
}

Previous discussion

Discussion based on the last proposal, which now has less relevance.

Quotes are from @Merovius.

A value with a generic method can not be assigned to any or a generic interface type (question: Why single out any here? I can also type-assert an io.Reader to have an Identity method. I don't understand what makes any special here). That is, it is not allowed to write HasIdentity(v) or any(v) (and question: Is it allowed to write io.Reader(v), assuming v also has a Read method with the right signature? If so, why is this allowed but not any?)

You've got a lot packed in here. Assuming you mean v := p1.S{}, you might be asking the following:

1. Q: Why can't we typecast to any(v) or HasIdentity(v) directly? A: Because I wanted an explicit type listing for generic method instantiations first. The issue with any is that it strips any binding between the generic type and the generic interface that we need for compile time method instantiation.

2. Q: Assume p1.S also has a Read method that satisfies the io.Reader interface. Can I write io.Reader(v)? (aka Why single out any?) A: You don't need to instantiate any generic methods to use io.Reader, so at first glance it looks OK. But it can get problematic if you then use io.Reader as a type proxy for any: that is, if you later typecast it back over to a generic interface. Disallowing that entirely would be overly restrictive, but it would force us to be up front about parameter types for method instantiation. It's the same problem with going direct to any: see previous question.

I think the revised proposal helps us here:

func CheckSIdentityWithoutAdoption() {
  v := p1.S{}

  x := io.Reader(v)       // assuming p1.S has a Read method
  y := x.(p2.HasIdentity) // PANIC: x does not adopt generic interface p2.HasIdentity
  z := any(x)
  p3.CheckIdentity(z)
}

func CheckSIdentityWithAdoption() {
  v := adopt(p1.S{}, p2.HasIdentity)

  x := io.Reader(v)       // assuming p1.S has a Read method
  y := x.(p2.HasIdentity) // OK, because {x via v} has adopted p2.HasIdentity
  z := any(x)
  p3.CheckIdentity(z)     // no problem there either
}
  

You don't address what I brought up here. That is, you talk about how conversions to and from interfaces work. But even without interfaces, generic methods create problems, as they de-facto allow to pass around uninstantiated generic functions. We don't allow that currently, because it puts unacceptable constraints on the implementation strategy.

Your cited comment is pretty cryptic to me, but from your recent meetup talk, it sounds like you're referring to this: given a generic function like func F[T any](v T) T { return v }, the statement f := F is a compile time error, but the statement f := F[int] is OK.

For the "No parameterized methods" example, f := (p1.S{}).Identity would be the equivalent first expression, and I have no problem with that being erroneous. That is, I don't think we need uninstantiated generic methods. What would you do with it?

I'd want the expression f := p1.S{}.Identity[int] to be OK: it is analagous to the non-generic version, where it behaves like a closure over the receiver p1.S{} for the Identity method specialized to int. Same thing if you define var s p1.S: erroneous use on f := s.Identify, and OK on f := s.Identify[int].

Structs having fields that use type parameters are a straightforward extension: you just list the types up front, e.g. v := R[int]{} for some struct R having a type parameter on a field, and access the methods in the same way as above.

What about interfaces? The expression f := (HasIdentity[int](p1.S{})).Identity looks OK to me, since it has also been speciailized to int, while simultaneously instantiating the method for that specialization. This syntax follows the "concrete interface" ideas of my last post. A syntax that is closer to the preceeding discussion would be f := (HasIdentity(p1.S{})).Identity[int].

Most of the apparatus you introduce doesn't seem to really have a point. If a function has to type-assert a generic interface down to its concrete instantiation to call the method - why even accept a generic interface? Why would you write func CheckIdentityTyped(v p2.HasIdentity) if you could also just write func CheckIdentityTyped(v p2.HasIdentity[int]) if that's the instantiation you are ultimately interested in?

This reads like an uncharitable criticism of the sample, rather than seeing it as a vehicle to illustrate ideas so we can progress this GH issue. Nonetheless, you might want to pass the interface because you want to have a type switch that treats several specialized types differently. e.g. int vs. string vs. something else.

If you want to use p2.HasIdentity[int] as an argument type, that's OK with me. That just means that you can then make method calls without further type checks (your next question).

It seems incredibly cumbersome to have to type-assert every method call. A lot of the use-cases for generic methods come down to chaining them. This wouldn't be possible, as every method call would have to be interceded with type-assertions. I don't think this is satisfying in actual usage.

The typecast would only be necessary for a returned generic interface. A non-interface generic type would not need type assertions along the chain. Recall that my interest is not with chained interface calls, but I agree that it should not be cumbersome.

It would be interesting to see if we could just infer the interface types for a chain, and I think you can: see the additional example below. This means that in the code example of my last post, there's no reason to disallow the following calls,

v.Identity[int](0) // ???INVALID??? interface is not concrete
v.Identity(0)      // ???INVALID??? as above, with inferred type paramete

they can be treated as the unprotected type-asserted call.

Generic interface chaining sample code

type Store interface[T, OutT any] {
	Log() Store
	Filter(func(T) bool) Store
	Map(func(T) OutT) Store
	Slice() []T
}

type storeImpl[T any] struct {
	v []T
}

func NewStore[T any](v ...T) Store {
	return storeImpl[T]{
		v: v,
	}
}

func (s storeImpl[T]) Log() Store {
	for _, el := range s.v {
		fmt.Println(el)
	}
	return s
}

func (s storeImpl[T]) Filter(f func(v T) bool) Store {
	var out []T
	for _, el := range s.v {
		if f(el) {
			out = append(out, el)
		}	
	}	
	return storeImpl[T]{v: out}
}	

func (s storeImpl[T]) Map[OutT any](f func(v T) OutT) Store {
	var out []OutT
	for _, el := range s.v {
		out = append(out, f(el))
	}
	return NewStore[OutT](out...)
}

func (s storeImpl[T]) Slice() []T {
	return s.v
}

func Usage() {
	var s Store = Store[int|string, string](NewStore(1,2,3,4))

	s
		.(Store[int, string])
			.Log()
		.(Store[int, string])
			.Filter(func(v int) bool { return v%2 == 0 })
		.(Store[int, string])
			.Map[string](func(v int) string { return strconv.Itoa(v) } )
		.(Store[string, string])
			.Log()
}

// Some of the types in Usage() above are don't care: let's replace them with
// an underscore and see how that looks...
func UsageWithDontCareTypes() {
	var s Store = Store[int|string, string](NewStore(1,2,3,4))

	s
		.(Store[_, _])
			.Log()
		.(Store[int, _])
			.Filter(func(v int) bool { return v%2 == 0 })
		.(Store[int, string])
			.Map[string](func(v int) string { return strconv.Itoa(v) } )
		.(Store[string, _])
			.Log()
}

// This syntax moves the types into each method call, where the type
// parameters match the interface method definition from left to right.
// 
// Alternatively, we could write the generic interface like
// StoreExplicitOrdering where the type parameters are explicitly ordered on
// each method.
func UsageWithPerCallTypes() {
	var s Store = Store[int|string, string](NewStore(1,2,3,4))

	s
		.Log()
		.Filter[int](func(v int) bool { return v%2 == 0 })
		.Map[int, string](func(v int) string { return strconv.Itoa(v) } )
		.Log()
}

// See comment in above about this syntax. It is a bit verbose...
type StoreExplicitOrdering interface[T, OutT any] {
	Log() Store
	Filter[T](func(T) bool) Store
	Map[T, OutT](func(T) OutT) Store
	Slice[T]() []T
}

// Perhaps we can also just infer the types...
func UsageWithInferredTypes() {
	var s Store = Store[int|string, string](NewStore(1,2,3,4))

	s
		// No type parameters (no type-specific method instantiation required).
		.Log() 
		// Can infer T == int from the func arg type.
		.Filter(func(v int) bool { return v%2 == 0 }) 
		// Can infer T == int, and OutT == string from the func arg type.
		.Map(func(v int) string { return strconv.Itoa(v) } ) 
		// No type parameters, as above
		.Log() 
}

The problem I mention here also exists for your suggestion. Seemingly harmless, backwards compatible changes in a package can now cause any program including it to crash. With no ability to detect it. That's because you still suggest the same semantics for interface type-assertions, even if you use a different syntax for "instantiations of a method".

There are differences:

1. Here, generic method types can't be used with an interface unless it becomes an instantiated generic type through a concrete interface. (Constraint 2.) This forces the developer to nominate types on the call graph, and not OOB like your scenario.

2. The above linkage can be used at compile time to ensure parameterized method instantiations are complete for all generic interface call sites.

   Consider my first example, with package p4 replaced in its entirety:

   package p4
   
   import (
     "p1"
     "p2"
     "p3"
   )
   
   func CheckSIdentity() {
     p3.CheckIdentity(p2.HasIdentity[string](p1.S{})) // concrete interface typecast
   }
   

   We know by the concrete interface typecast that p1.S implements p2.HasIdentity for string. But in CheckIdentity, we need p2.HasIdentity[int], and there is no such concrete interface typecast. If type assertions in CheckIdentity weren't conditional, the compiler could flag an error to prevent a run time panic. This is a big difference: there is no inadvertent crash here.

   Better still would be to simplify the proposal so the developer don't have to list the types explicitly. We can then drop this idea of a concrete interface, and get the compiler to determine the set of parameterized types automatically. That's the basis for the revised proposal...

[ianlancetaylor]

You are proposing large and significant language changes, which make generic methods behave significantly different from non-generic methods. The benefit of these changes appears to be to permit call chaining. I don't think the benefit outweighs the cost.

Note that scanning the AST does not work in the general case where we dynamically assign values to a variable of type any and, for example, send that variable on a channel. Some other package with no import relationship may receive that variable and do a type assertion on it. No form of AST scanning will permit discovering all such cases.

[jpap]

You are proposing large and significant language changes, which make generic methods behave significantly different from non-generic methods. The benefit of these changes appears to be to permit call chaining. I don't think the benefit outweighs the cost.

Thanks for considering the proposal, @ianlancetaylor. Just in case there was a misunderstanding here: the proposed language addition is limited exclusively to how generic interface methods would behave relative to their non-generic cousin; there are no changes to the language for non-interface and non-generic types. I also hope you disregarded my first proposal; the second has a much smaller surface area.

Support for generics beyond functions can be split into two parts:

1. Support for types with parameterized methods.
2. Support for generic interfaces that work with Part 1.

I haven't seen disagreement here on the importance and utility of Part 1, despite your generics design suggesting that "...it's much less clear why we need methods at all. If we disregard interfaces, any parameterized method can be implemented as a parameterized function." It's unfortunate that in lieu of non-interface type methods and chaining, we are left with polluting a package namespace with faux top level "methods". I'd like to help improve on this at the very least.

Given that one of your suggestions from the same generics design was that "we could decide that parameterized methods do not, in fact, implement interfaces", then perhaps we can consider moving forward with Part 1, and separately leave Part 2 unimplemented, or at least until we can agree on how to redefine generic interfaces, as your design goes on to say: "So while parameterized methods seem clearly useful at first glance, we would have to decide what they mean and how to implement that." I tried to make progress on exactly that, but perhaps there is no easy or cheap way to do that without also making some changes to the language?

How do you feel about separating these parts out so we can make progress?

Note that scanning the AST does not work in the general case where we dynamically assign values to a variable of type any and, for example, send that variable on a channel. Some other package with no import relationship may receive that variable and do a type assertion on it. No form of AST scanning will permit discovering all such cases.

I'm not sure I follow you here: if we are scanning the AST from package main when performing a build, recursively across all imports^†, how are we not going to traverse the type assertion on the far end of a channel as you have described?

---

^† Obvious optimizations aside, where per-package scan results can reside in the build artifact cache.

[Merovius]

@jpap

I haven't seen disagreement here on the importance and utility of Part 1, despite your generics design suggesting that […]

I think the point is not that there is no utility, but that the utility seems too small to justify the cost.

If we had a solution to the question of how to pass around uninstantiated generic functions, one "cost" of generic methods is mostly eliminated - we can give reasonable, consistent answers to questions around interface satisfaction and the like under those circumstances. At the same time, the benefit is high, as it can also cover the (IMO by far most common) use-case of functional APIs like Functor.

Without a solution to that, limiting generic methods to be purely syntactic sugar for generic functions - if they can't be used to implement interfaces, they can't appear in interfaces, they can't be accessed using reflect or interface type-assertions - both the benefit is lower and the cost is higher (as there is an inconsistency between generic and non-generic methods in their behavior).

The benefit is not zero. But the tradeoff gets significantly worse.

[atdiar]

@jpap [edit] Sorry, nvm.. it actually just works already. I fail to see why we would need uninstantiated parametric methods on instantiated or non-generic types. I must be missing something.

How would stream be defined in:

func (si *stream[IN]) Map[OUT any](f func(IN) OUT) stream[OUT] ?

(was thinking of something more like type stream[IN any, OUT any] struct{...} which can be defined already but I'm realizing that won't help)

Seems that a function makes sense here if IN and OUT are actually unrelated i.e. Mapping to any type should be possible on any instance of stream. But then no chaining...

Was way off the mark... Need to ponder on it.

[jpap]

@ianlancetaylor, if you wish to avoid additions to the language, or splitting types and interfaces into two phases, should we then propose implementation strategies exclusively?

Here is one that I hope you might consider:

1. While the compiler constructs the AST, we can separately record a lists of: A) type names having parameterized methods, B) interfaces names, and C) type names at what I will call "consumption sites", as recorded in AST nodes FuncDecl, TypeAssertExpr, ChanType, etc.

   • We don't need type information here yet.
   • This can be done independently per package, and the results can be serizlied into a package artifact that sits in the build cache or the package object file.

2. Before link, where type information is available, we can combine the above lists (across all packages) to determine generic method instantiations:

   • Resolve C to Bs.
   • Match As to Bs. For each unique type-set, instantiate the method(s) for each A.

This is essentially the implementation part of my last proposal, without the language additions and where the generic interfaces behave in the same way as their non-generic cousins (i.e. no more explicit "opt-in"; think of it instead as every type value is implicitly "opt-in").

This is not much more expensive than my last proposal: the "Match" step involves more generic methods to compare, but i) it can be parallelized, and ii) short-circuits would quickly cull the search space. I'd expect that search space to be relatively small in any case: a ballpark in the low thousands of generic types for a very large project? It would still be cheaper than analyzing the call graph, made more difficult by the channel scenario you raised.

While it might be tempting to avoid this and do it at run time, JIT compilation (aside from warmup and complexity) is a non-starter on platforms that have code signing restrictions (e.g. iOS, or hardened runtime macOS) that disallow anything JIT-like. Even a fallback interpreter can still run afoul of App Store rules. Whatever we can do at compile time would be a win.

[Merovius]

Just to put more spanners into the works: For better or worse, plugin exists. So the compiler does not, in general, know all the code that is compiled into a Go program.

[ianlancetaylor]

@jpap As I wrote in the proposal document: "[E]ven that traversal is not sufficient in the general case when type reflection gets involved, as reflection might look up methods based on strings input by the user. So in general instantiating parameterized methods in the linker might require instantiating every parameterized method for every possible type argument, which seems untenable."

[jpap]

@ianlancetaylor I addressed that previously, where I talk about ammending the laws of reflection to rule-out run time instantiation of methods that aren't already part of the program's compile time structure. That aligns with an earlier comment where @mccolljr highlights that the current reflect package can't represent a generic interface for an arbitrary type-set at run time.

If reflect currently does not allow us to create reflect.Types for generic interfaces over a given set of type parameters at run time, nor allows us to instantiate generic functions in the same way, why do we have to do something different for generic methods? Reflection today only allows us to work with the program's compile time structure.

@mccolljr's suggestion in another comment is similar to mine: let's add API to inspect the program's compile time structure, and allow us to work with whatever instantiations are available.

I'd love to understand your thoughts on this distinction.

If progressing this issue is hinged on run time reflect calls for arbitrary instantiations, are you open to having the compiler emit a slower fallback path for that purpose via boxing? Would we need to support a way to do the same for generic functions as well?

Given the "every possible type argument" approach is untenable, and JIT is a non-starter on some platforms even ignoring its stated drawbacks, how else can we move this forward?

[ianlancetaylor]

In general the reflect package is suppose to permit any operation that is permitted in the language. There are at present a few missing operations, but that is regarded as a bug. It would be strange if the language permits type conversions that the reflect package does not permit. We shouldn't lock that in.

It's true that the reflect package doesn't permit instantiating generic functions. That's OK, because today there is no way to express such a function. You can only refer to an instantiated function, not to an uninstantiated one. If we adopt some form of generic methods, that may no longer be true.

It's important that we not introduce non-orthogonal restrictions. Where there are restrictions, we want those restrictions to be as simple as possible. The restriction "methods may not have type parameters" is simple and easy to understand, even if some people don't like it. The restriction "the reflect package may refer to a instantiation of a generic method if and only if that instantiation occurs somewhere in the program" is not easy to understand, and may change depending on which packages are imported.

Given the "every possible type argument" approach is untenable, and JIT is a non-starter on some platforms even ignoring its stated drawbacks, how else can we move this forward?

I've been thinking about this problem off and on for about four years and have not yet found a way to move it forward. That doesn't mean that it can't be done, of course.

[jpap]

Thanks for conveying your thought process @ianlancetaylor. It definately helps, as it's now pretty clear that this feature hinges on reflection support...

I've been thinking about this problem off and on for about four years and have not yet found a way to move it forward.

Have you considered including an instantiation for each method of interest that uses the Dictionaries approach (i.e. not vanilla or GC Shape Stenciling; I called it boxing in my last message) in addition to the usual instantiations that would be generated by the compiler, and where the dictionary can be constructed by reflect at run time based on the actual type parameters provided?

Construction of the dictionary likely requires additional information needed for each type parameter, stashed away in a read only segment. All of this can of course be omitted when the reflect package is not being used.

This approach would have a small dictionary set up cost, but it would survive in a JIT-less environment and could be cached for subsequent calls. I doubt that would be problematic in practice, given that the reflect-based path is not known for its speed.

[aarzilli]

Dictionaries is how generics are implemented currently. It doesn't solve the problem. Even if you could instantiate dictionaries at runtime, each parametric method would have to be instantiated for every possible combination of gc shapes, it's less than every possible combination of types but still too large.

[jpap]

@aarzilli, I'm not proposing to use GC Shapes. I'm sorry if my last message was unclear: I was proposing to use the pure Dictionaries approach; see the link in my last message.

[aarzilli]

You still need to do codegen to instantiate a dictionary: func (s *S) Equals[T any](a, b T) { return a == b } needs an equality function for whatever type is passed.

[jpap]

You still need to do codegen to instantiate a dictionary: func (s *S) Equals[T any](a, b T) { return a == b } needs an equality function for whatever type is passed.

That's good to know. Why is that something that cannot be performed at run time, given the type information? It doesn't have to run at the same speed: I'd imagine it is like performing a recursive memcmp with holes (sparse) and pointer following over a structure-like memory blob.

EDIT: i.e. we can pass a run time general Equals func into the dictionary, and have an additional field that specifies the type T so it knows what to do at run time. Or just modify the pure Dictionary scheme to suit.

[Merovius]

@jpap It's not that it can't be done at runtime. It's that it has to be done at runtime and we don't want to prescribe a particular implementation strategy. That's one of the design criteria for Go generics. If we abandoned that and where fine requiring a dynamic, boxing approach, the problem would go away (as we'd know how to implement passing around uninstantiated generic functions).

[Merovius]

(That's FWIW, why suggesting a specific implementation strategy won't really help. We'd need a way to describe how to spec this out so we don't need a specific implementation strategy)

[jpap]

@Merovius, perhaps my last suggestion wasn't so clear, as @aarzilli was thrown off by it too. I make an effort to trim my comments before posting, sometimes at the expense of clarity. I apologize for that, so here is a detailed follow up:

1. I am suggesting that the current generic instantiation approach remain as-is, expanded further to support types with parameterized methods using the approach described here.

2. Only when package reflect is being used, I am suggesting that we also include one additional instantiation per type-parameterized method based on the pure dictionary approach. That is, each method gets the usual instantiations (Note 1, above), and just one more that is specifically optimized for reflect so it can construct the dictionary and execute, at run time, with arbitrary types that satisfy any given parameterized type constraints.

   This proposal can resolve the blocker that @ianlancetaylor alluded to, and can allow us to progress this GH issue. This choice does not impact how generics are implemented outside of reflect, nor does it prevent us from changing the approach used with reflect in the future. That is, I am not trying to unify the reflect and non-reflect use cases, that may otherwise overconstrain such a "unified implementation strategy".

   We can use any known implementation strategy on the relfect path: pure dictionary (boxing), GC Shapes, or pure stenciling. The latter two require some kind of JIT compiler or (byte code) interpreter; the first does not. Each weighs differently on the executable size. The latter two also require us to ship a JIT compiler and/or interpreter that works from an IR of the "method body". The first is the simplest, and just includes the method body in the text section of the executable as usual.

   The current generics implementation has chosen GC Shapes from the same set of implementation strategies. That generates a plurality of instantiations that aims to be smaller than pure stenciling. It cleverly tries to balance run time overhead vs. executable size.

   This proposal is no different, where we choose a pure dictionary approach optimized for the run time use case of reflect. We include just one more adjacent instantiation into the existing plurality. This choice also tries to balance run time overhead vs. executable size: there's no JIT compiler or interpreter to ship in the executable, and instead of the method body IR, we include only one reflect-optimized instantiation of the method itself. This choice has a further advantage, in that it naturally works with platforms having code signing restrictions.

Together, this proposal does not constrain the existing generics design in any way. Note 1 above "permits different implementation choices" as it always has. Note 2 shows we have the same flexibility. What guides us to one of many implementation choices here are other constraints, for example wanting to avoid JIT compilation or interpretation, and overhead considerations. The pure dictionary approach is a good one when used in the way proposed.

Put another way, can we change the implementation approach and keep the design the same? It's safe to answer "Yes" for Note 1. I hope I've demonstrated that the answer remains "Yes" for Note 2 also.

At the end of the day, we've got to choose an implementation. It dictates what code needs to be written so we can move this GH issue forward. A good language design (here, generics design) will allow us to change that later if needed, and without developers rewriting their Go code. Unless there is some technical reason otherwise, I think my proposal enables that while simultaneously allowing for other pragmatic constraints. If I have missed anything in that respect, I'd appreciate a collaborative discussion so we can figure out how to move this forward together.

---

On UGFs, I did address the topic previously but that discussion ended there. I outlined how can fit into my proposal and how it just appears to be a special case. I'm not going to comment on that further here because I see it as tangential. I appreciate that you may see it differently, and perhaps as part of an overall language and/or implementation design, which might be worth focusing on as a separate GH issue in its own right.

[ianlancetaylor]

we also include one additional instantiation per type-parameterized method based on the pure dictionary approach.

This does not give implementations the freedom to implement generics as they see fit: it prohibits implementations from using the pure stenciling approach used by most C++ implementations. They would be required to also implement the pure dictionary approach. That requirement currently does not exist.

Note that the pure dictionary approach is not straightforward, certainly much less straightforward than stenciling. See the questions and caveats in the design doc you linked to. There are also some real performance concerns: certain kinds of code will likely run significantly slower than other very similar code.

[AaronFriel]

As I understood it, the dictionary would only be chosen when an implementation uses generics to access what would today be an uninstantiated version of a generic function.

Given that what is being proposed is today impossible, I don't follow the discussion of unreasonable performance, unless you're interpreting that cost as being imposed on current (1.18 valid) programs?

[ianlancetaylor]

What I mean is that when the compiler can see the type arguments for an instantiation, it will use stenciling or a gc-shape dictionary. When it can't see the type arguments, it will use the pure dictionary form, which I think will tend to be significantly slower. But whether the compiler can or can't see the type arguments can be a subtle distinction, depending on inlining and devirtualization and other optimizations.

[jpap]

[Quotes below from @ianlancetaylor]

we also include one additional instantiation per type-parameterized method based on the pure dictionary approach.

This does not give implementations the freedom to implement generics as they see fit: it prohibits implementations from using the pure stenciling approach used by most C++ implementations. They would be required to also implement the pure dictionary approach. That requirement currently does not exist.

This was proposed for reflect code paths only (see Note 2). Non-reflect code paths would operate using the current GC Shapes approach, and there's no reason why pure stenciling can't be used there instead (see Note 1).

It sounds like you want both the reflect and non-reflect code paths to share the same implementation strategy? That will of course limit your implementation choices, forcing you down the run time code generation path, for which you've already said "would be very complex to implement, and would be surprisingly slow at run time". Other than performance, addressed below, why must they be unified?

Note that the pure dictionary approach is not straightforward, certainly much less straightforward than stenciling. See the questions and caveats in the design doc you linked to.

I appreciate that aspects of the current dictionary design doc could be more concrete, but is that maturity a function of the Go core team pursing the competing GC Shapes proposal, rather than the dictionary design itself having one or more show-stoppers?

Many of the questions posed in that doc have candidate solutions and ideas that can be pursued. There might be some work to fully specify a reflection-specific dictionary design, but I don't see any obvious show-stoppers.

@randall77: I see you've authored that doc. I apologize for the abrupt mention, but I'd love to get your insight on this. Do you think a dictionary design tailored to calling generic methods at run time from the reflect package is feasible?

There are also some real performance concerns: certain kinds of code will likely run significantly slower than other very similar code.

We already have generic code that can run signficiantly slower than its "manually monomorphic" equivalent.

Here we are talking about the difference in performance of a generic method call under reflection as compared to a similar non-reflection path. That difference might end up in the (sub?) microsecond range, which of course could get amplified by a generic method making further generic calls within a loop. Given that it is well-known that reflect has a performance impact and performance critical code should avoid it where possible, is this really a problem in practice? By taking the non-reflect path for such critical operations, we don't have to give up full performance.

[ianlancetaylor]

It's a good point that I'm assuming that this new approach will be required for cases that do not use the reflect package. That's because I don't believe that it's feasible to analyze the entire program at link time in order to discover all instantiations, as you've suggested above. That would be a heavy link time cost, especially considering that the linker is already the slowest part of a typical build.

And, that said, yes, I do prefer that all execution paths use the same implementation strategy. It's costly, and likely error-prone, to have two different ways of compiling a generic function. That is a lot of compiler code to write and test. And of course one version of the compiler code will be used less often, and therefore get less testing. Keeping it all up to date is a lot of work.

These are real costs, and they should have a commensurate benefit. If the only benefit is the ability is to chain method calls, that doesn't seem like a good cost/benefit tradeoff to me.

I agree that you've found a path that I didn't explicitly mention in the generics proposal doc. But I do think it runs afoul of one of my comments: "We believe that this design permits different implementation choices." Your suggestion does not permit different implementation choices: it requires a specific implementation choice for some cases.

And all language design choices are cost/benefit decisions. We can't just focus on the benefits. We have to weigh the costs.

[randall77]

And, that said, yes, I do prefer that all execution paths use the same implementation strategy. It's costly, and likely error-prone, to have two different ways of compiling a generic function.

+++1 to that. The generics implementation is already enormously complicated, I'd hate to have 2 of them.

I appreciate that aspects of the current dictionary design doc could be more concrete, but is that maturity a function of the Go core team pursing the competing GC Shapes proposal, rather than the dictionary design itself having one or more show-stoppers?

It is both.

To my mind the biggest obstacle here is how stack data are handled. The pure dictionary design doc hints at that issue, but the more I think about it the more messy it gets. We need to match the calling convention to the fact that generic code needs to access all generic values by reference instead of by value. It affects the calling convention, particularly with regabi. We might need to generate wrappers everywhere. We also need to either implement an alloca mechanism for values of generic type, or have special in-the-heap-but-treated-by-GC-as-on-the-stack allocations.

We'd also need generate a pure dictionary body for every function (method?) in case it is called by reflect, even though that would be very rare. And because it might happen, we'd need all the code in the binary even though at runtime it never gets used. Or we need some way to encode generic Go function bodies into the binary that can be JITed somehow... Any way you slice it, it seems like an enormous lift for comparatively little gain.

[mccolljr]

These are real costs, and they should have a commensurate benefit. If the only benefit is the ability is to chain method calls, that doesn't seem like a good cost/benefit tradeoff to me.

With respect, I find this statement too reductive.

I suspect that the emoji reactions and heavy engagement with this proposal are not because developers have a really strong desire to "chain methods". The benefit is one of making certain patterns that are (apparently) common among programs written in Go more concise and approachable. This of course includes cases of method chaining in use cases such as pipeline transformations, but it also enables a more concise and accurate way of describing the set of types that another type can act upon. If I want to express that a type has a method whose argument can be any integer type, my current options are:

1. Define a unique method for every supported integral type,
2. Define a global generic function with an appropriately constrained parameter,
3. Accept an any value and type-switch
4. Force users to wrap integer types in an interface which I or they must implement

These solutions all require some tradeoff between compile-time validation, runtime speed, and ease of use, and none are exactly the same as a generic method.

Many people in the Go community were attracted to this language because they felt it gave them concise ways to express complex concepts (channels, goroutines). Many do not write Go as if it were C code. Many would like to bring patterns like reactive programming to Go. Many of those same people find the current state of affairs in Go unfriendly to those patterns.

To categorize the benefit of this feature as "method chaining" seems like it is at best a failure to understand the community's desires here, and at worst an intentional slight.

And all language design choices are cost/benefit decisions. We can't just focus on the benefits. We have to weigh the costs.

This is certainly true any anyone who has worked on a large and complex project can probably remember at least one instance of pain caused by selecting the wrong tradeoff. At the same time, it seems there is unwillingness to compromise with the community with regards to this issue.

As an example - it seems to me that a number of us would be comfortable with a solution that is not (immediately) supported by reflect. Even the current reflect package has limitations, both related and unrelated to generics. The fact that this is a sticking point for this proposal feels inconsistent with impact reflect compatibility has had on historical decision making.

[atdiar]

As far as I'm concerned, I don't find the statement too reductive wrt the explanations that have been given. This looks very non-trivial to implement for a minor gain in ergonomy.

Maybe there exists a way to chain function calls that does not rely on uninstantiated generic code, if that's just what's missing?

What are some alternatives?

[jpap]

[Quotes below from @ianlancetaylor]

... I don't believe that it's feasible to analyze the entire program at link time in order to discover all instantiations, as you've suggested above. That would be a heavy link time cost, especially considering that the linker is already the slowest part of a typical build.

I think we are talking about the "Match step" for interfaces here, and I must respectfully disagree. :) As previously outlined, that can be parallelized and short-circuits would quickly cull the search space. The subsequent instantiations would of course add time toward a build, but there is no reason why they cannot also reside in the build artifact cache. These instantiations are unlikely to change drastically in the usual develop-build-run cycle.

Where a Go program cannot have generic methods, and instead relies on faux top level "methods", the linker is still going to perform instantiations of some kind. This set of instantiations might be smaller for a typical program, but the pre-link work doesn't magically disappear.

These are real costs, and they should have a commensurate benefit. If the only benefit is the ability is to chain method calls, that doesn't seem like a good cost/benefit tradeoff to me.

The other benefit is having a more expressive type system that supports generics as a natural extension of the existing object model, without developers having to resort to top level functions. Not only does that workaround pollute a package namespace, it makes the object model less intuitive to the package user. That has other impacts on DX, such as making the code harder to read, and harder for IDEs to propose code-completions.

I'm sure that as generic functions become more popular throughout the Go community, the desire for generic methods will grow. I'm sure there'll be developers who will be surprised to learn that we can't have generic methods, even though we have generic functions today. I can appreciate the complexity behind the implementation, but many developers might not care, and especially when other languages like C++, Java, and C# do support generic methods.

I agree that you've found a path that I didn't explicitly mention in the generics proposal doc. But I do think it runs afoul of one of my comments: "We believe that this design permits different implementation choices." Your suggestion does not permit different implementation choices: it requires a specific implementation choice for some cases.

Stricly speaking the design doesn't limit choice, as I previously outlined in Note 2: there is nothing stopping us from implementing a JIT compiler and/or interpreter. We might choose not to, but the design by no means restricts it. Perhaps it's a subtle point, but in that sense my suggestion to choose Pure Dictionaries is not much different to the Go team having chosen GC Shapes.

[Quotes below from @randall77]

We need to match the calling convention to the fact that generic code needs to access all generic values by reference instead of by value. It affects the calling convention, particularly with regabi. We might need to generate wrappers everywhere.

I appreciate your insight here. Given we are limiting the pure pure dictionary instantiations to what I'd call the "reflection silo", we can dictate the calling convention (ABI0) because the only call sites would be from reflect, or another pure dictionary instantiation in the same silo. This might require some thought for a closure cb produced within a generic method that is passed to a non-generic function in the "non-reflection silo", but in that case the parameters are non-generic by definition and so perhaps the calling convention for cb is not so critical?

For the same reason, I don't see a need for wrappers everywhere, because the call sites not part of the pure dictionary approach are in their own silo. Within the reflection silo, each call site is effectively the wrapper.

We also need to either implement an alloca mechanism for values of generic type, or have special in-the-heap-but-treated-by-GC-as-on-the-stack allocations.

I like the idea you proposed in the design doc where the offsets are fully described in the dictionary so the instantiation remains simple and compact: those dictionary entries could be computed and memoized at run time by reflect based on the type information available to it (for a given set of type parameters), supported by a dictionary that is formed from the function/method body at compile time. Call sites within the body would need similar treatment, and can be done by chaining back to reflect as a psuedo run time?

We'd also need generate a pure dictionary body for every function (method?) in case it is called by reflect, even though that would be very rare. And because it might happen, we'd need all the code in the binary even though at runtime it never gets used. Or we need some way to encode generic Go function bodies into the binary that can be JITed somehow... Any way you slice it, it seems like an enormous lift for comparatively little gain.

This is a great argument in support of changing the laws of reflection when it comes to generic methods. The only reason I have been pursuing a solution to this is because @ianlancetaylor hinted that such support would be necessary to progress. I totally agree that the benefit on the reflection use-case is unwarranted, so why not just constrain its use?

Perhaps this topic can be reconsidered, through the lens of a cost/benefit decision. Wouldn't it be nice to have generic method support for the common case where it is not called from reflect?

---

As I edit this post, I see some similar commentry from @mccolljr. I won't remove the overlapping sentiment here, but I am grateful to others for sharing their independent view.

[Merovius]

@mccolljr

To categorize the benefit of this feature as "method chaining" seems like it is at best a failure to understand the community's desires here, and at worst an intentional slight.

I believe you are misreading Ian here. If by "this feature" you mean "allowing type parameters in methods" (i.e. the broadest interpretation of this issue), it is indeed untrue that chaining is not the only benefit. But the sentence you quoted is referring specifically to a narrower interpretation of this issue - the one we can achieve without requiring a particular implementation strategy. That specific subset of "this feature" is the one which Ian says only confers the benefit of method chaining.

[randall77]

I appreciate your insight here. Given we are limiting the pure pure dictionary instantiations to what I'd call the "reflection silo", we can dictate the calling convention (ABI0) because the only call sites would be from reflect, or another pure dictionary instantiation in the same silo. This might require some thought for a closure cb produced within a generic method that is passed to a non-generic function in the "non-reflection silo", but in that case the parameters are non-generic by definition and so perhaps the calling convention for cb is not so critical?

I don't understand what a "silo" is here. Call sites in question can be anywhere an indirect call happens in the program. The method might be generatable only by reflect, but you could .Interface().(func(int)float64) or whatever on it and pass it anywhere. The caller's only option is to call it with the standard Go calling convention. If the receiver needs something else, there needs to be a wrapper.

Wrappers aren't the end of the world. We already use them for several cases. But we'd need a lot more for this feature.

I like the idea you proposed in the design doc where the offsets are fully described in the dictionary so the instantiation remains simple and compact: those dictionary entries could be computed and memoized at run time by reflect based on the type information available to it (for a given set of type parameters),

Right.

supported by a dictionary that is formed from the function/method body at compile time.

I'm not sure what you mean here. We'd need what the design doc calls a proto-dictionary encoded in the binary somewhere, and reflect would have to instantiate that proto-dictionary using the type arguments supplied at run time. That would get you a concrete dictionary you could then use in the method body.

Call sites within the body would need similar treatment, and can be done by chaining back to reflect as a psuedo run time?

This may not be necessary, if the dictionary built above can be instantiated recursively, i.e. if we can figure out all the subdictionaries required once we know the instantiating types (which would normally be doable, except for weird nonmonomorphizable cases which cause infinite regress.).

Perhaps this topic can be reconsidered, through the lens of a cost/benefit decision. Wouldn't it be nice to have generic method support for the common case where it is not called from reflect?

I think it would be really nice to have type parameters on methods.

But there's a lot of cost involved in getting there. Cost in both the complexity of the language and in the complexity of the implementation.

[jpap]

[Quotes below from @randall77]

I don't understand what a "silo" is here.

The set of method instantiations generated for reflect vs non-reflect use cases. I described them as a silo because I figured code executing in the latter does not call the former. Your comment below is a good counterexample for that.

Call sites in question can be anywhere an indirect call happens in the program. The method might be generatable only by reflect, but you could .Interface().(func(int)float64) or whatever on it and pass it anywhere. The caller's only option is to call it with the standard Go calling convention. If the receiver needs something else, there needs to be a wrapper.

That makes sense. For a method, my understanding is that the above results in a call to the trampoline reflect.methodValueCall. If that's the case, could we add a flag to reflect.methodValue to indicate that we need to use a special calling convention into the (pure dictionary) method instantiation? Do we need a wrapper in that case?

... supported by a dictionary that is formed from the function/method body at compile time.

I'm not sure what you mean here. We'd need what the design doc calls a proto-dictionary encoded in the binary somewhere, and reflect would have to instantiate that proto-dictionary using the type arguments supplied at run time. That would get you a concrete dictionary you could then use in the method body.

That's right: the "dictionary" here is just metadata that provides the size and offsets in the stack frame for generic locals that are needed within the method body. My apologies: I should have just called it "metadata" to disambiguate it from the dictionary used to perform a generic call.

Under "Stack Layout" in your design doc, the example would be local var z in func f. That is, we would put together such metadata that "looks into a method" at compile time to assist reflect construct the dictionary at run time.

Call sites within the body would need similar treatment, and can be done by chaining back to reflect as a psuedo run time?

This may not be necessary, if the dictionary built above can be instantiated recursively, i.e. if we can figure out all the subdictionaries required once we know the instantiating types (which would normally be doable, except for weird nonmonomorphizable cases which cause infinite regress.).

Makes sense. I was thinking that it might be more complex to be looking through call graphs, and simpler to just take it one jump at a time.

I think it would be really nice to have type parameters on methods.

But there's a lot of cost involved in getting there. Cost in both the complexity of the language and in the complexity of the implementation.

The additions to the language naturally follow the existing generic syntax for functions. I agree that there is a bunch of work on the implementation, which we can significantly reduce by amending the laws of reflection and/or constrain what is possible with the reflect package.

So long as there is one or more implementation path(s) for full run time support (which I think there are), we can always weigh the additional implementation cost for full run time support, if there is sufficient interest in supporting reflect at a later date.

[ianlancetaylor]

Where a Go program cannot have generic methods, and instead relies on faux top level "methods", the linker is still going to perform instantiations of some kind. This set of instantiations might be smaller for a typical program, but the pre-link work doesn't magically disappear.

With the current scheme used by the gc compiler, the only step required by the linker is to discard duplicate instantiations, which is cheap. The actual instantiations are created by the compiler. The difference is that the compiler can run in parallel across packages, where the linker runs only once when creating the final executable.

[jpap]

[Quote below from @ianlancetaylor]

With the current scheme used by the gc compiler, the only step required by the linker is to discard duplicate instantiations, which is cheap. The actual instantiations are created by the compiler. The difference is that the compiler can run in parallel across packages, where the linker runs only once when creating the final executable.

Thank you for the clarification. I see that is discussed in more detail in the pure stenciling design doc under "Risks".

For the proposed scheme, each independent compiler invocation can of course instantiate generic methods for direct calls, where duplicates are discarded during the final link. As such, there is no impact on build time when compared the "top level faux method" pattern.

That is distinct from the separate pre-link^† procedure previously described, which strictly speaking is only needed for generic method instantiations via an interface. The procedure can be parallelized and may benefit from having its own object-like format for use with the build cache.

In summary, there are three types of generic instantiations: functions (existing; in compiler), methods called directly (new; in compiler), and methods called via an interface (new; pre-link).

We might separately consider unifying all three, so that all generic instantiations are performed in a single pre-link procedure. I can appreciate the reluctance toward a large design change, but the benefit here could be a net performance gain as we can avoid production of duplicates. All major parts are parallelizable, and caching can speed up subsequent builds. Another benefit is that the final unified implementation may be simpler to maintain.

---

† Pre-link means as implemented in the go tool before invoking the linker, or as part of the linker itself.

[AaronFriel]

I'm curious about the focus on reflect here. Reflect today lacks comprehensive support for generic types. Reflection lacks support for inspecting generic type arguments, given a concrete type Foo[Bar], one cannot obtain a reflect.Type instance for Bar.

Two points:

1. Would treating generic methods similarly to generic types negatively impact users? e.g.: if generic methods are simply invisible to reflection.
2. If generic types and generic methods allow users to not need reflect, isn't that a virtuous goal in itself?

For the use case I'm concerned with, I believe generic types and methods in the language would obviate the need to use reflect. We wouldn't need it, because we could accomplish all we need with methods. And given how panic-prone reflection APIs are, we'd greatly prefer the compile time safety of working entirely in generic methods (even if we had to fall back to dynamic dispatch via vtables/dictionaries!) over cumbersome reflect based framework implementations.

[atdiar]

I may be wrong but I think that reflect allows to create composites of already known types. I imagine that these types could be used as type arguments to generic type constructors eventually.

But then, since there might have not been any instantiation at compile time of these novel types, there is no way to pattern match signatures/ use the dictionary and the vtable needs to filled up with the new entries at runtime. It could require an additional level of indirection when dispatching in the case of generic methods.

That's the way I understood it.

[Merovius]

@AaronFriel Note that the reason generic types are not part of reflect is that generic types don't exist in a valid Go program. For them to be represented in reflect, you'd need to be able to pass a variable of uninstantiated generic type to reflect.TypeOf. So, their lack in reflect directly mirrors their unavailability in the Go type system.

Also, I don't really think reflect itself is the issue. reflect is, if anything, a stand-in for the general idea of runtime representation of types. The same issues exist without reflect, just using interface type-assertions.

Would treating generic methods similarly to generic types negatively impact users? e.g.: if generic methods are simply invisible to reflection.

AIUI, Ian said clearly that we don't want to get locked into that.