Github actions e2e tests with istio-ambient are very flaky

[ReToCode]

Examples

• https://github.com/knative/serving/actions/runs/6889846778
• https://github.com/knative/serving/actions/runs/6871768680

Also take a look at the discussion here: https://github.com/knative/serving/pull/14610#issuecomment-1893894692

[dprotaso]

@nak3 @ReToCode did anything change recently with the ambient configuration?

[dprotaso]

Maybe Istio 1.20?

https://github.com/knative/serving/pull/14629

[nak3]

I'm sure the flake started before 1.20. I bumped 1.20 if the flake can be fixed or not after getting a ping on slack https://cloud-native.slack.com/archives/C04LMU0AX60/p1700061890467089

The change may be affected is bumping to v1.9.3 https://github.com/knative-extensions/net-istio/pull/1194 https://github.com/knative-extensions/net-istio/pull/1196

[nak3]

Today's github action does not hit the flake at all.

• Serving repo - https://github.com/knative/serving/actions/runs/6934043757

I have tried 3 times on my repo but no error happened.

• https://github.com/nak3/serving/actions/runs/6938581702
• https://github.com/nak3/serving/actions/runs/6938840029
• https://github.com/nak3/serving/actions/runs/6938839999

It seems that infra (github action) level issue?

[ReToCode]

Well that is interesting. @nak3 your environment with GH actions should not be different, right? So I have no idea why this works well and then doesn't like in https://github.com/knative/serving/pull/14627.

[nak3]

Well that is interesting. @nak3 your environment with GH actions should not be different, right?

Yes, no different and actually it failed yesterday.

[ReToCode]

Hm, so still disabling it until we figure it out?

[nak3]

Yeah, I'm ok with disabling it.