ci test flake: Access Log in memory Access Logs Grpc [It] can stream access logs

[jenshu]

Gloo Edge Version

1.11.x (beta)

Kubernetes Version

No response

Describe the bug

Access Log in memory Access Logs Grpc [It] can stream access logs 
Step #10 - "test": /workspace/gloo/test/e2e/access_log_test.go:141
Step #10 - "test": 
Step #10 - "test":   Expected
Step #10 - "test":       <string>: 
Step #10 - "test":   to equal
Step #10 - "test":       <string>: local-test-upstream-102_default
Step #10 - "test": 
Step #10 - "test":   /workspace/gloo/test/e2e/access_log_test.go:172
Step #10 - "test": 
Step #10 - "test":   Full Stack Trace
Step #10 - "test":   github.com/solo-io/gloo/test/e2e_test.glob..func1.1.3.4.3()
Step #10 - "test":   	/workspace/gloo/test/e2e/access_log_test.go:172 +0xcc8
Step #10 - "test":   github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc000bb2480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:113 +0xfd
Step #10 - "test":   github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc000bb2480, 0x4e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:64 +0x185
Step #10 - "test":   github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00099e320, 0x74b0180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/it_node.go:26 +0xba
Step #10 - "test":   github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc000ac43c0, 0x0, 0x74b0180, 0xc0002aa500)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:215 +0x7fe
Step #10 - "test":   github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc000ac43c0, 0x74b0180, 0xc0002aa500)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:138 +0x188
Step #10 - "test":   github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc000a882c0, 0xc000ac43c0, 0x0)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:200 +0x17c
Step #10 - "test":   github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc000a882c0, 0x1)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:170 +0x236
Step #10 - "test":   github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc000a882c0, 0x8)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:66 +0x146
Step #10 - "test":   github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0002a89a0, 0x7f60e838be68, 0xc000c8aa80, 0x685004b, 0x9, 0xc000fdabe0, 0x2, 0x2, 0x7548658, 0xc0002aa500, ...)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/internal/suite/suite.go:79 +0x83a
Step #10 - "test":   github.com/onsi/ginkgo.runSpecsWithCustomReporters(0x74bd560, 0xc000c8aa80, 0x685004b, 0x9, 0xc000fdabc0, 0x2, 0x2, 0x2)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:245 +0x358
Step #10 - "test":   github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters(0x74bd560, 0xc000c8aa80, 0x685004b, 0x9, 0xc000fdabc0, 0x1, 0x2, 0x1)
Step #10 - "test":   	/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:228 +0x1eb
Step #10 - "test":   github.com/solo-io/gloo/test/e2e_test.TestE2e(0xc000c8aa80)
Step #10 - "test":   	/workspace/gloo/test/e2e/e2e_suite_test.go:55 +0x298
Step #10 - "test":   testing.tRunner(0xc000c8aa80, 0x6983860)
Step #10 - "test":   	/usr/local/go/src/testing/testing.go:1193 +0x203
Step #10 - "test":   created by testing.(*T).Run
Step #10 - "test":   	/usr/local/go/src/testing/testing.go:1238 +0x5d8

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=f4cad27c-b323-45db-b6ff-4b68043917b5

Steps to reproduce the bug

run build-bot tests in ci

Expected Behavior

should pass

Additional Context

No response

[jenshu]

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=96c6b55b-2476-4f3c-ad46-40e456175dc1

[jenshu]

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=7813f51e-27e7-4f1e-923d-09336ecd9d9a

[jenshu]

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=ddce7f4a-e272-4ca6-89ef-26dc9d3e969f

[elcasteel]

I stepped through this while looking at another test flake and the thing that sticks out to me is that we start up a server and do not have any checks to verify that it is up and running before we do the test: https://github.com/solo-io/gloo/blob/master/test/e2e/access_log_test.go#L325 I haven't been able to reproduce the flake locally to verify and I'm not sure what a good check for server liveness would be.

[mkazin]

https://storage.googleapis.com/solo-public-build-logs/log-21f6849a-ce2d-423b-9d66-ee38890ada8b.txt

[nfuden]

this happend 4 times here is one of them https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=d10a98af-57a3-488c-b1bc-e087b19da9ff

[elcasteel]

This was fixed by: https://github.com/solo-io/gloo/pull/6448

[elcasteel]

Still seeing this: https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=5a04f531-70f4-4bcd-ac45-dcae481a7af9

[jenshu]

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=b7793848-b00b-4fd1-838e-7e2f99c30757