Add retry to pytest for transient failure

[shunping]

This change enables a retry mechanism for pytest to improve the developer experience and the reliability of our test/release pipelines.

Recently, we have observed an increase in flaky Python tests. These failures disrupt development and release validation, often requiring manual and time-consuming test reruns. Some post-commit workflows can take hours to complete, and multiple flaky tests can significantly delay getting a green build.

To address this, this change leverages the pytest-rerunfailures plugin with the following configuration:

• --reruns 1: Allows for a single retry of a failed test. We want to keep this a small number so that we do not risk hiding a truly flaky test.
• --reruns-delay 5: Introduces a 5-second delay before the retry.

This approach provides a balance between mitigating transient failures (such as GRPC deadline exceeded) and ensuring that consistently flaky tests are still surfaced and addressed.

fixes #37038

[gemini-code-assist[bot]]

Summary of Changes

Hello @shunping, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust retry mechanism for pytest tests across various Python environments. By integrating the pytest-rerunfailures plugin and configuring it within tox.ini, the system will now automatically re-run failed tests up to three times with a short delay. This enhancement is designed to significantly improve the reliability of CI/CD pipelines by addressing and reducing failures caused by intermittent or flaky tests, ensuring more consistent and accurate test results.

Highlights

• New Dependency Added: The pytest-rerunfailures plugin has been added as a new dependency in sdks/python/setup.py. This plugin enables the automatic re-execution of failed pytest tests.
• Pytest Retry Mechanism Implemented: The tox.ini configuration has been updated across multiple test environments to utilize the newly added pytest-rerunfailures plugin. Pytest commands will now automatically retry failed tests 3 times with a 5-second delay between retries.
• Improved Test Stability: This change aims to enhance the stability of the test suite by mitigating issues caused by transient or flaky test failures, reducing the need for manual re-runs of CI/CD jobs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

[^1]: Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

[shunping]

cc'ed @damccorm

[github-actions[bot]]

Assigning reviewers:

R: @damccorm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions[bot]]

Reminder, please take a look at this pr: @damccorm

[shunping]

I agree to close this for now and focus on resolving test flakiness itself.