sentry-cocoa icon indicating copy to clipboard operation
sentry-cocoa copied to clipboard

Fix or disable flaky SentryHTTPTransport flush tests

Open philipphofmann opened this issue 6 months ago • 1 comments

Description

The SentryHTTPTransport tests for the flush implementation sometimes flake as here https://github.com/getsentry/sentry-cocoa/actions/runs/15700976571/job/44235777418

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("3") is not equal to ("0")

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("SentryFlushResult(rawValue: 2)") is not equal to ("SentryFlushResult(rawValue: 0)") - Flush should not time out.

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("2") is not equal to ("0")

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("SentryFlushResult(rawValue: 2)") is not equal to ("SentryFlushResult(rawValue: 0)") - Flush should not time out.

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("1") is not equal to ("0")

testFlush_CallingFlushDirectlyAfterCapture_Flushes, XCTAssertEqual failed: ("SentryFlushResult(rawValue: 1)") is not equal to ("SentryFlushResult(rawValue: 0)") - Flush should not time out.

Ideally, we should fix the flaky tests and not disable them, because they test the flush functionality. It's worth mentioning that the ANR tests were also flaky due to the TestCurrentDateProvider not being thread-safe. After we fixed this problem with https://github.com/getsentry/sentry-cocoa/pull/5299 the tests are more stable now. So when investigating this problem keep in mind that the test code could actually have bugs or maybe the flush implementation is buggy.

philipphofmann avatar Jun 17 '25 08:06 philipphofmann

From Cocoa Sync 06/18:

We need to look into the proposed changes of https://github.com/getsentry/sentry-cocoa/pull/5220 again, but @philipphofmann believes that a test plan will not help and instead we need to evaluate the tests themselves again.

philprime avatar Jun 18 '25 13:06 philprime