go101
Panic with generics on protobuf messages
Hi,
I'm getting the following issue on lates (v0.7.8) and various variants of it on previous versions when running golds ./... on our (close source) repo:
1.18+.go:326: Message
panic: should not
goroutine 3 [running]:
go101.org/golds/code.(*CodeAnalyzer).comfirmDirectSelectorsForInstantiatedType(0x140000be008, 0x1402a943a20, 0x11a, 0x1401ef97bc8, 0x1401ef97b98)
/Users/user/go/pkg/mod/go101.org/[email protected]/code/1.18+.go:327 +0xde4
go101.org/golds/code.(*CodeAnalyzer).comfirmDirectSelectorsForInstantiatedTypes(0x140000be008)
/Users/user/go/pkg/mod/go101.org/[email protected]/code/code-analyse.go:2813 +0x194
go101.org/golds/code.(*CodeAnalyzer).AnalyzePackages(0x140000be008, 0x1401ef97e10)
/Users/user/go/pkg/mod/go101.org/[email protected]/code/code-analyse.go:59 +0x1c4
go101.org/golds/internal/server.(*docServer).analyze(0x1400008a700, {0x14000132060, 0x1, 0x1}, {{0x1050ea2c8, 0x6}, {0x1400000e035, 0xb}, 0x1, 0x1, ...}, ...)
/Users/user/go/pkg/mod/go101.org/[email protected]/internal/server/server.go:366 +0x48c
go101.org/golds/internal/server.Run.func1()
/Users/user/go/pkg/mod/go101.org/[email protected]/internal/server/server.go:139 +0x78
created by go101.org/golds/internal/server.Run in goroutine 1
/Users/user/go/pkg/mod/go101.org/[email protected]/internal/server/server.go:138 +0x458
Sorry, line numbers may be offset as I instrumented the code to see a bit better what happens.
After some investigations, the smallest snippet I could find to reproduce is something like this:
package main
import (
"google.golang.org/protobuf/proto"
)
type ProtoContent struct {
Proto proto.Message
}
// ProtoMessage interface is used for generics and represents a pointer on proto.Message.
// For various cases, T cannot be used directly as proto Structs are pointer receivers,
// and that does not work well with generics.
type ProtoMessage[T any] interface {
proto.Message
*T
}
// This is the definition which makes the crash happen.
func MakeFoo[T proto.Message, SubT ProtoMessage[T]](t T) ProtoContent {
return ProtoContent{Proto: t}
}
func main() {
}
We found it very hard to work with generics and protobuf 😅
Same exact thing here, though we're not using protobuf. (Though third-party dependencies use them, so maybe it doesn't matter)
I just pushed a new version v0.7.9, which walks around this problem.
Since Go added custom generics, it becomes much harder to find all implementation relations, mainly for the relations involving generic instantiated types.
The problem is hard to solve and I have not enough spare time to study it deeply now. I'm sorry that I can't make a perfect solution.
@TapirLiu There's nothing to apologize for. :) External dependencies evolve with or without our say.
Thanks for iterating on it so quickly! It still panics for one of my projects — are there any places I could add debug output to help triage? I'm happy to poke around if there are areas you think are likely. It has ~20k LOC with mixed generic use, so I'm not sure exactly what may be causing the issues.
Stacktrace
> golds version
Golds v0.8.0
> golds ./...
[Analyzing] Start analyzing ...
[Analyzing] Preparation done: 39.124417ms
[Analyzing] 2 files parsed: 682.916µs
[Analyzing] 4 files parsed: 4.272827958s
[Analyzing] 8 files parsed: 4.272944208s
[Analyzing] 16 files parsed: 4.273236583s
[Analyzing] 32 files parsed: 4.273598208s
[Analyzing] 64 files parsed: 4.274452625s
[Analyzing] 128 files parsed: 4.276269375s
[Analyzing] 256 files parsed: 4.285417583s
[Analyzing] 512 files parsed: 4.406472416s
[Analyzing] 1024 files parsed: 4.472450541s
[Analyzing] 2048 files parsed: 4.636970041s
[Analyzing] 4096 files parsed: 5.1371745s
[Analyzing] All 4625 files are parsed: 5.59381375s
[Analyzing] Collected 970 packages: 362.34675ms
[Analyzing] Collected 132 modules: 586.615791ms
[Analyzing] Sorted packages by dependency relations: 570.375µs
[Analyzing] Collected source files: 1.119167ms
[Analyzing] Collected declarations: 64.188125ms
panic: runtime error: index out of range [1] with length 1
goroutine 17 [running]:
go101.org/golds/code.transformTypeArgs(...)
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/code/1.18+.go:372
go101.org/golds/code.(*CodeAnalyzer).comfirmDirectSelectorsForInstantiatedType(0x140000b8008, 0x1402bdfe2c0, 0x6d3, 0x140496ebbc8, 0x140496ebb98)
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/code/1.18+.go:161 +0xc60
go101.org/golds/code.(*CodeAnalyzer).comfirmDirectSelectorsForInstantiatedTypes(0x140000b8008)
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/code/code-analyse.go:2813 +0x194
go101.org/golds/code.(*CodeAnalyzer).AnalyzePackages(0x140000b8008, 0x140496ebe10)
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/code/code-analyse.go:59 +0x1c4
go101.org/golds/internal/server.(*docServer).analyze(0x140042028c0, {0x1400001e170, 0x1, 0x1}, {{0x100aadc24, 0x6}, {0x1400000e055, 0xb}, 0x0, 0x0, ...}, ...)
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/internal/server/server.go:366 +0x48c
go101.org/golds/internal/server.Run.func1()
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/internal/server/server.go:139 +0x78
created by go101.org/golds/internal/server.Run in goroutine 1
/Users/coxley/.go/pkg/mod/go101.org/[email protected]/internal/server/server.go:138 +0x458
Aha! I've narrowed it down to the culprit dependency. It's actually a bit surprising since I've tested it working with (IMO) more complex generics code.
Repo: https://github.com/orsinium-labs/enum
Reproduction Steps:
cd $(mktemp -d)
git clone https://github.com/orsinium-labs/enum \
&& cd enum \
&& golds -silent ./...
I've narrowed it down to the smallest set of types that trigger the panic:
package foo
type Member[T comparable] struct {
Value T
}
type Equaler[V comparable] interface {
Equal(other V) bool
comparable
}
type Enum[M Member[V], V comparable] struct {
members []M
v2m map[V]*M
}
func Parse[M Member[V], V Equaler[V]](e Enum[M, V], value V) *M {
for v, m := range e.v2m {
if v.Equal(value) {
return m
}
}
return nil
}
Specifically, it seems to really not like self-referential types. The following doesn't panic, but causes golds to hang indefinitely. This is in a single package, with a single file.
package foo
type Equaler[V comparable] interface {
Equal(other V) bool
comparable
}
func Equal[V Equaler[V]](a, b V) bool {
return a.Equal(b)
}
// > tree -a
// .
// ├── go.mod
// └── foo.go
//
// > golds -silent ./...
// [Analyzing] Start analyzing ...
// [Analyzing] Preparation done: 52.417875ms
// [Analyzing] 2 files parsed: 1.639083ms
// [Analyzing] 4 files parsed: 33.702375ms
// [Analyzing] 8 files parsed: 144.608208ms
// [Analyzing] 16 files parsed: 144.928ms
// [Analyzing] 32 files parsed: 145.208125ms
// [Analyzing] 64 files parsed: 145.941541ms
// [Analyzing] 128 files parsed: 153.178666ms
// [Analyzing] 256 files parsed: 159.140791ms
// [Analyzing] All 275 files are parsed: 241.48025ms
// [Analyzing] Collected 28 packages: 447.243583ms
// [Analyzing] Collected 2 modules: 26.900542ms
// [Analyzing] Sorted packages by dependency relations: 33.375µs
// [Analyzing] Collected source files: 80.833µs
// [Analyzing] Collected declarations: 2.930875ms
// -- hangs indefinitely here --
Pushed v0.8.1 to walk around the two problems. It is intended to walk around the first one, but happens to also walk around the second one. I really don't have a clear understanding on many complexities introduced by custom generics now.