Staged transformation test flake

[jenshu]

Which tests failed?

getting a lot of these this week:

Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;9m[FAIL][0m [0mStaged Transformation [38;5;243mno auth [0mcan enable enhanced logging [38;5;9m[1m[It] staged logging enabled, individual logging enabled[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;243m/workspace/gloo/test/e2e/staged_transformation_test.go:644[0m

https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=c638bbf4-d95b-4ffd-be27-f0d22a86a8e1

Initial Investigation

checking if it fails on 1.16 https://github.com/solo-io/gloo/pull/9290

Additional Information

No response

[jenshu]

tests passed on 1.16

[jbohanon]

Analyzing build logs it looks like this flake first started appearing on 3/25

[nfuden]

Its odd because in the linked logs the 4th failure looks for a substring which it cant find but DOES EXIST!