[Bug]: Test breakage detection tool does not handle tests consistently timing out and cancelled

[Abacn]

What happened?

It is found that

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml

has been timing out since Oct 30, but wasn't detected by the tool (it should reopen #34990)

~relevant code path:~

~https://github.com/apache/beam/blob/8ff2c940321cc8adb49e2fc590a1d35fd7113add/.test-infra/tools/flaky_test_detection.py#L83~

EDIT:

actually the alert is fetched from grafana. We should change the alert criteria there

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• [ ] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [x] Component: Infrastructure
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner