code generation hanging, busy-loop

[eitanb]

i've just downloaded github.com/globusdigital/deep-copy v0.5.4 trying to generate code for a simple type, it worked as expected. so sanity check is fine. however, my real code has a complex graph of types (with multiple cycles like A -> B -> C -> A) & running the tool on these causes it to run for minutes. at some point (> 5 minutes), i killed it.

retrying is the same, so its not a one-time failure.

is there any way to get verbose log so that i can maybe see where is it looping? Thanks

[romshark]

Can you please provide the exact code for reproduction?

[eitanb]

i cant point the finger at exactly what is the issue - trying smaller samples doesn't show the same issue. i also cannot share the original code (for now, checking on this...) so i'm trying to look at your code :)

i looked at the walkType() function: hacking the code a little, its obvious that there are some type-cycles that cause an infinite recursion in this function. looking at the source & sink parameters, they seems to get longer & longer (by name) & the same path's of members is seen on stack over & over again.

for such a cycle, there is no way to have a proper order of generating the code. i think the issue to tackle this is that when you generate code for type A with a member type B, use the B.DeepCopy() when you know you are going to generate a code for it later on. you seem to check whether the DeepCopy exist using the Go package API but it only tells you whether the DeepCopy() existed when you loaded the module, i.e.: BEFORE you started this run.

any set of types for which this current run is about to generate code will be seen as having no such API. but it should call that API to easily break cycles doing so manually is too complicated for an order of 100 types with lots of references.

[eitanb]

to overcome the issue, i've generated code for some types & used --skip to NOT follow the graph. once a few of these were generated the large code-generation managed to run fine.

so this is a work-around. but it also supports the suggested issue/fix i mentioned in the previous reply.

[egawata]

The following code can reproduce the problem easily.

package main

import (
	"net/http"
)

type Foo struct {
	name string
	req  *http.Request
}

then try deep-copy against the struct Foo. I tried it, and it hadn't been finished after 10 minutes wait on my machine.

$ deep-copy -o t_dp.go --type Foo .

We can spot the field causing deep recursion by adding the following code.

diff --git a/main.go b/main.go
index 3b9f45c..37bd471 100644
--- a/main.go
+++ b/main.go
@@ -336,6 +336,9 @@ func walkType(source, sink, x string, m types.Type, w io.Writer, imports map[str
        }

        depth++
+       if depth > 100 {
+               log.Fatalf("deep recurtion. depth = %d, sink = %s", depth, sink)
+       }
        under := m.Underlying()
        switch v := under.(type) {
        case *types.Struct:

2022/09/23 12:12:15 deep recurtion. depth = 101, sink = cp.req.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.Response.Request.TLS.PeerCertificates[i94].Issuer.Names[i98].Type[i100]

Any advisory warning like this would help us, so that we could add --skip option to the structs/fields.

[urandom]

yeah, this might be a good idea. perhaps we can add a flag to specify the max depth and print a warning once reached, and stop descending.

@egawata would you ble able to create a pull request for this?

[ivanjaros]

yeah, this might be a good idea. perhaps we can add a flag to specify the max depth and print a warning once reached, and stop descending.

@egawata would you ble able to create a pull request for this?

depth is not the problem but recursion. the code should not process types it has already processed. it should keep each type in memory and re-use the template it generated. that way you cannot get stuck in infinite loop like https://github.com/globusdigital/deep-copy/issues/23