firebase-cpp-sdk icon indicating copy to clipboard operation
firebase-cpp-sdk copied to clipboard

Add FLAKY_TEST_SECTION_RESET and use it to reset FCM in between flakes.

Open jonsimantov opened this issue 2 years ago • 1 comments

Description

Provide details of the change, and generalize the change in the PR title above.

When GetToken fails, it seems like it will usually keep failing unless FCM is reinitialized. This change allows us to reinitialize FCM between attempts, to try to get GetToken to get a token.

This change adds an extra flaky test macro, FLAKY_TEST_SECTION_RESET, that can be put in the middle of a FLAKY_TEST_SECTION_BEGIN and ..._END pair. If there is a FLAKY_TEST_SECTION_RESET, this code will be run after each flaky failure, before the retry, and can be used to reset or reinitialize things before the test runs again.

(Unfortunately, this doesn't seem to 100% fix the problem in TestReceiveToken, which may be caused by an underlying issue with the SDK. Here is an example of it still being broken. 🙁)

It does seem to fix the issues with SendMessageToToken and SendMessageToTopic. However, when retrying SendMessageToToken, we can get into a situation where it receives the message from the previous attempt; add some logic to recognize this situation and consume the extra message. (Here is a test run where this occurred before this fix.)


Testing

Describe how you've tested these changes. Link any manually triggered Integration tests or CPP binary SDK Packaging Github Action workflows, if applicable.

Will run the Android integration tests a few times and analyze the logs to see if the flaky test triggers and if the reinitialize helps.


Type of Change

Place an x the applicable box:

  • [ ] Bug fix. Add the issue # below if applicable.
  • [ ] New feature. A non-breaking change which adds functionality.
  • [ ] Other, such as a build process or documentation change.

Notes

  • Bug fixes and feature changes require an update to the Release Notes section of release_build_files/readme.md.
  • Read the contribution guidelines CONTRIBUTING.md.
  • Changes to the public API require an internal API review. If you'd like to help us make Firebase APIs better, please propose your change in a feature request so that we can discuss it together.

jonsimantov avatar Feb 02 '23 18:02 jonsimantov

Integration test with FLAKINESS (succeeded after retry)

Requested by @jonsimantov on commit refs/pull/1201/merge Last updated: Thu Feb 2 15:34 PST 2023 View integration test log & download artifacts

Failures Configs
messaging [TEST] [FLAKINESS] [Android] [1/3 os: windows] [1/4 android_device: android_target]
(1 failed tests)  FirebaseMessagingTest.TestSendMessageToToken

Add flaky tests to go/fpl-cpp-flake-tracker

github-actions[bot] avatar Feb 02 '23 18:02 github-actions[bot]