firebase-ios-sdk icon indicating copy to clipboard operation
firebase-ios-sdk copied to clipboard

[Crashlyics] Address some Crashlytics flakes and disable others in nightly tests

Open paulb777 opened this issue 1 week ago • 3 comments

This PR should stabilize the Crashlytics unit tests after asking gemini to find and address several flaky tests. This PR doesn't touch the library itself other than exposing an additional property to the tests.

It also disables the flaky FIRCLSSettingsTests.m in nightly test runs.

This is a fresh start on #15551 which went too far with adding unnecessary churn and disabling some functionality.

This is an example of the common flake that will be suppressed in the nightly tests:

Screenshot 2025-12-01 at 8 55 57 PM

#no-changelog

Gemini's summary:

Summary of Changes

This pull request focuses on enhancing the stability and reliability of the Crashlytics unit test suite. It implements targeted fixes for existing flaky tests, such as improving file system interactions and cleanup procedures. Additionally, it introduces a configurable exclusion for certain flaky tests during nightly builds, aiming to prevent intermittent failures in automated pipelines. The changes also include minor refactoring to improve test code maintainability and thread safety in mock objects.

Highlights

  • Test Stability Improvements: Several Crashlytics unit tests have been updated to address flakiness, ensuring more reliable test results by improving file system interactions and cleanup procedures.
  • Flaky Test Exclusion: A mechanism has been introduced to conditionally exclude known flaky tests (specifically FIRCrashlyticsReportTests.m) from nightly test runs, preventing build failures due to intermittent issues.
  • Mock File Manager Thread Safety: The mock file manager used in tests now includes thread-safe operations, which can help prevent race conditions in concurrent test environments.
Changelog
  • Crashlytics/UnitTests/FIRCLSContextManagerTests.m
    • Ensures the root directory for the mock file manager is explicitly created during test setup.
  • Crashlytics/UnitTests/FIRCLSExistingReportManagerTests.m
    • Modified setUp to ensure a clean state by recreating the file manager's root directory after removal.
  • Crashlytics/UnitTests/FIRCLSReportUploaderTests.m
    • Implemented a polling loop with a timeout to wait for file removal, making the test more resilient to timing variations.
  • Crashlytics/UnitTests/FIRCLSUserDefaultsTests.m
    • Added explicit removal of test-specific keys from NSUserDefaults in tearDown to prevent test interference.
  • Crashlytics/UnitTests/Mocks/FIRCLSMockFileManager.m
    • Added @synchronized blocks to all methods that access or modify the internal fileSystemDict to ensure thread safety.
  • Package.swift
    • Refactored the Crashlytics unit test target definition into a new function crashlyticsUnitTestChooser() to allow conditional exclusion of FIRCrashlyticsReportTests.m based on the CRASHLYTICS_NIGHTLY environment variable.

paulb777 avatar Dec 02 '25 01:12 paulb777