firebase-cpp-sdk icon indicating copy to clipboard operation
firebase-cpp-sdk copied to clipboard

Published 20 hours ago • verified publisher icon firebase

[C++] Nightly Integration Testing Report for Firestore

Open github-actions[bot] opened this issue 3 years ago • 23 comments

❌  [build against repo] Integration test FAILED

Requested by @sunmou99 on commit 82350f13e0bcc3ab9e5693a29255325f1a247e83 Last updated: Wed Sep 21 05:46 PDT 2022 View integration test log & download artifacts

Failures Configs
missing_log [TEST] [ERROR] [Android] [1/3 os: ubuntu] [1/5 android_device: emulator_latest]
[TEST] [ERROR] [MacOS] [1/2 ssl_lib: arm64] [boringssl]
firestore
(5 items)[BUILD] [ERROR] [Linux] [1/2 ssl_lib: x86] [All 2 build_type]
[BUILD] [ERROR] [MacOS] [arm64] [boringssl]
[BUILD] [ERROR] [Windows] [x64, x86] [boringssl]
[BUILD] [ERROR] [Windows] [x64] [openssl]
[TEST] [FLAKINESS] [Android] [1/3 os: windows] [1/5 android_device: android_target]
(1 failed tests)  CRASH/TIMEOUT

Add flaky tests to go/fpl-cpp-flake-tracker

❌  [build against SDK] Integration test FAILED

Requested by @firebase-workflow-trigger[bot] on commit 82350f13e0bcc3ab9e5693a29255325f1a247e83 Last updated: Wed Sep 21 07:45 PDT 2022 View integration test log & download artifacts

Failures Configs
firestore [BUILD] [ERROR] [Linux] [1/2 ssl_lib: x86] [boringssl]
[TEST] [FLAKINESS] [iOS] [macos] [1/6 ios_device: ios_target]
(1 failed tests)  CRASH/TIMEOUT

Add flaky tests to go/fpl-cpp-flake-tracker

github-actions[bot] avatar Feb 02 '22 11:02 github-actions[bot]

CanStopListeningAfterTerminate flaked, which will be fixed once the Android dependencies are upgraded to a version that picks up the fix: https://github.com/firebase/firebase-android-sdk/pull/3374

dconeybe avatar Feb 02 '22 21:02 dconeybe

Closing again, since the test failures are still CanStopListeningAfterTerminate, which will be fixed once the Android dependencies are upgraded.

dconeybe avatar Feb 03 '22 15:02 dconeybe

CanStopListeningAfterTerminate again. Closing.

dconeybe avatar Feb 04 '22 15:02 dconeybe

One failure was CanStopListeningAfterTerminate. The other one was odd: it looked like gmscore got borked and took down a bunch of apps with it, including the Firestore test app. I'm going to ignore this flake for now since the logcat lacked useful information to help debug. If this persists after CanStopListeningAfterTerminate is fixed then investigation may be warranted.

dconeybe avatar Feb 07 '22 02:02 dconeybe

I won't be investigating these flakes until the fix for CanStopListeningAfterTerminate is incorporated, because the root bug that it fixes can have unpredictable side effects.

dconeybe avatar Feb 07 '22 15:02 dconeybe

As discussed, I'm just going to leave this ticket opened until the fix for CanStopListeningAfterTerminate from the Android SDK is pulled in.

dconeybe avatar Feb 08 '22 15:02 dconeybe

build-desktop-macos-latest-openssl failed very strangely: it's complaining that the contents of bundle.nanopb.cc (a file from the iOS SDK) are not valid UTF-8. According to the error message they look like garbage, possibly some binary file. I attempted a local build and it compiled successfully. I've pinged the FPL team about this.

build-android-macos-latest appears to have failed due to a network issue while gradle was downloading artifacts. This is probably a transient error and I'm going to ignore it for now.

test-android-ubuntu-latest-android_target failed due to CanStopListeningAfterTerminate timing out (b/205063697). The fix for this flake, https://github.com/firebase/firebase-android-sdk/pull/3374, went into 24.0.2 but this repository still depends on Firestore 24.0.1. So the fix for this flake won't be realized until the dependency is upgraded to 24.0.2.

test-ios-macos-latest-simulator_latest failed due to NumericTransformsTest.CreateDocumentWithIncrement flakily timing out. I've reported it but will ignore it for now.

dconeybe avatar Feb 28 '22 16:02 dconeybe

FYI The flaky test CanStopListeningAfterTerminate was fixed by #872, which upgraded Firestore's Android dependency from 24.0.1 to 24.1.0.

dconeybe avatar Mar 28 '22 00:03 dconeybe

build-desktop-windows-latest-openssl and build-desktop-windows-latest-boringssl failed due to some vcpkg download error. This is probably a transient error and I will ignore it for now.

test-android-macos-latest-emulator_32bit looks like the testapp either crashed or timed out while running BundleTest.LoadInvalidBundlesShouldFail. My gut instinct is that this particular test is just a red herring and there is something else going wrong. The logs are very vague though. I'll investigate.

dconeybe avatar Mar 28 '22 00:03 dconeybe

Update: This vcpkg error was fixed by updating the vcpkg submodule: https://github.com/firebase/firebase-cpp-sdk/pull/876

build-desktop-ubuntu-latest-boringssl and build-desktop-windows-latest-boringssl failed last night due to vcpkg downoad errors again:

Error: Failed to download from mirror set:
https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12175
https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404

dconeybe avatar Mar 29 '22 13:03 dconeybe

The test-android-windows-latest-emulator_latest has been failing for a few days now. It appears to be due to the tests timing out after 10 minutes.

Here is the log excerpt of interest, showing the 10-minute gap:

2022-03-31T11:37:18.6147880Z I0331 11:37:18.613914 4693212672 test_simulator.py:660] Running game-loop test: adb shell am instrument -w com.google.firebase.gameloop.test/androidx.test.runner.AndroidJUnitRunner
2022-03-31T11:47:43.0931380Z I0331 11:47:43.091164 4693212672 test_simulator.py:588] game-loop test error!!! reboot emualtor...

The logs from the tests do indeed show that the individual tests are taking a VERY long time. Here are some examples that show tests taking 5-10 seconds each, when each should generally take far less than 1 second:

2022-03-31T12:09:29.1550160Z [       OK ] FirestoreIntegrationTest.CanBeUsedWithReferenceValues (5041 ms)
2022-03-31T12:09:29.1654100Z [       OK ] FirestoreIntegrationTest.TestDocumentSnapshotEventsForDelete (3109 ms)
2022-03-31T12:09:29.1527290Z [       OK ] BundleTest.LoadInvalidBundlesShouldFail (3498 ms)
2022-03-31T11:52:37.5378910Z [       OK ] FirestoreIntegrationTest.TestDeleteDocument (3775 ms)

test-android-ubuntu-latest-emulator_latest also took a really long time.

Maybe the GitHub runners are indeed really slow and the 10-minute timeout is too aggressive. 🤷

dconeybe avatar Mar 31 '22 13:03 dconeybe

In test-android-windows-latest-emulator_target the failed test was QueryNetworkTest.EnableDisableNetwork. It failed because line 98 in TestQueriesFireFromCacheWhenOfflineImpl() timed out: QuerySnapshot snapshot = accumulator.AwaitServerEvent(); I'm going to assume that this is a transient failure since I've never seen this failure before.

In test-ios-macos-latest-simulator_latest the failed test was NumericTransformsTest.CreateDocumentWithIncrement. It also timed out waiting for a server response. I'm going to assume this is a flake as well.

dconeybe avatar Apr 05 '22 14:04 dconeybe

test-ios-macos-latest-simulator_min failed due to NumericTransformsTest.CreateDocumentWithIncrement flaking AGAIN (a 3rd time). I'm going to ignore it this one last time, but it will probably come up again.

dconeybe avatar Apr 10 '22 02:04 dconeybe

test-desktop-ubuntu-latest-boringssl failed due to ServerTimestampTest.TestServerTimestampsCanRetainPreviousValueThroughConsecutiveUpdates, which looks like a timeout flake. I'm going to ignore this for now.

dconeybe avatar Apr 11 '22 13:04 dconeybe

Hmm... Android build error (e.g. https://github.com/firebase/firebase-cpp-sdk/runs/6340472701).

> Could not resolve all task dependencies for configuration ':debugRuntimeClasspath'.
   > Could not resolve io.grpc:grpc-core:[1.44.1].
     Required by:
         project : > com.google.firebase:firebase-firestore:24.1.2 > io.grpc:grpc-android:1.44.1
         project : > com.google.firebase:firebase-firestore:24.1.2 > io.grpc:grpc-okhttp:1.44.1
      > Failed to list versions for io.grpc:grpc-core.
         > Unable to load Maven meta-data from https://jcenter.bintray.com/io/grpc/grpc-core/maven-metadata.xml.
            > Could not get resource 'https://jcenter.bintray.com/io/grpc/grpc-core/maven-metadata.xml'.
               > Could not GET 'https://jcenter.bintray.com/io/grpc/grpc-core/maven-metadata.xml'.
                  > org.apache.http.client.ClientProtocolException (no error message)

Update: Should be fixed by https://github.com/firebase/firebase-cpp-sdk/pull/938, which replaces jcenter with mavenCentral.

dconeybe avatar May 09 '22 18:05 dconeybe

All of the integration test failures reported at https://github.com/firebase/firebase-cpp-sdk/actions/runs/2408275729 are due to the Firestore Emulator failing to launch. This will be fixed by https://github.com/firebase/firebase-cpp-sdk/pull/975, which upgrades the java version installed on the GitHub Actions runners to be at least the minimum version required by the Firestore emulator.

dconeybe avatar May 30 '22 15:05 dconeybe

Integration test build errors will be fixed by https://github.com/firebase/firebase-cpp-sdk/pull/981. All of the build errors are on desktop platforms with linker errors like this:

undefined reference to `re2::RE2::~RE2()'

e.g. https://github.com/firebase/firebase-cpp-sdk/actions/runs/2434421755

This was caused by https://github.com/firebase/firebase-cpp-sdk/pull/977, which upgraded the iOS dependencies to 9.1.0, which added some new external dependencies (e.g. re2) which need to have the packaging step updated, and some additional cmake system dependencies on windows builds. Both of these will be fixed by https://github.com/firebase/firebase-cpp-sdk/pull/981.

dconeybe avatar Jun 03 '22 16:06 dconeybe

Also, test-ios-macos-12-ios_min failed with the following error:

ERROR: (gcloud.beta.firebase.test.ios.run) '11.4' is not a valid OS version

e.g. https://github.com/firebase/firebase-cpp-sdk/runs/6726407464

This should be fixed by updating the iOS versions: https://github.com/firebase/firebase-cpp-sdk/pull/982

dconeybe avatar Jun 03 '22 18:06 dconeybe

test-android-ubuntu-latest-emulator_target failed due to a timeout. There are no logs so it's impossible to tell what timed out. I did see this line near the end of the log, which may be why the logs are not present:

/Users/runner/work/_temp/93f9c3e5-c24d-4676-b3a9-ef248fcd218d.sh: line 2: [: missing `]'

build-desktop-macos-12-arm64-boringssl and build-desktop-macos-12-x64-boringssl failed with

unzip:  cannot find or open firebase_cpp_sdk.zip, firebase_cpp_sdk.zip.zip or firebase_cpp_sdk.zip.ZIP

dconeybe avatar Jun 06 '22 15:06 dconeybe

The flake of ValidationTest.FieldNamesMustNotBeEmpty in test-android-macos-12-android_latest-gameloop was due to a network flake. The failure of build-desktop-ubuntu-latest-x86-boringssl was due to a network flake as well.

dconeybe avatar Jun 29 '22 14:06 dconeybe

In https://github.com/firebase/firebase-cpp-sdk/actions/runs/2609287298 there were several failures:

  1. TransactionTest.TestMaxAttempts failed on several platforms. The error message suggests that calling Await() on the Future returned from DocumentReference.set() returned kFutureStatusInvalid. This seems odd. I've opened a PR to improve the error messages in case this happens again: https://github.com/firebase/firebase-cpp-sdk/pull/1012
  2. In test-ios-macos-12-simulator_min-gameloop NumericTransformsTest.CreateDocumentWithIncrement failed due to a timeout error. I'm going to ignore this as a flake for now.
  3. test-android-windows-latest-emulator_32bit-gameloop failed due to a timeout running Android's sdkmanager command, presumably due to a horribly slow internet connection. I'm ignoring it as an infrastructure-related flake.

In https://github.com/firebase/firebase-cpp-sdk/actions/runs/2609881077, the build-desktop-macos-12-arm64-boringssl action failed due to "unzip: cannot find or open firebase_cpp_sdk.zip, firebase_cpp_sdk.zip.zip or firebase_cpp_sdk.zip.ZIP". The non-Firestore nightly tests failed for the same reason: https://github.com/firebase/firebase-cpp-sdk/actions/runs/2609880259. There must be something going awry with the macOS arm64 builds. I'll ignore this since this will be investigated by others due to its non-Firestore-specific nature.

dconeybe avatar Jul 04 '22 16:07 dconeybe

The 4 test failures were all TransactionTest.TestMaxAttempts failing again with "Timeout waiting for doc.Set() to complete":

  1. test-desktop-ubuntu-latest-x86-openssl
  2. test-desktop-ubuntu-latest-x86-boringssl
  3. test-android-ubuntu-latest-emulator_32bit-gameloop
  4. test-android-windows-latest-emulator_32bit-gameloop

The logging improvement to this test to help diagnose (https://github.com/firebase/firebase-cpp-sdk/pull/1012) failed due to exceeding Windows path lengths. I'll remove the part of that PR that causes the path length problem and get it merged.

dconeybe avatar Jul 05 '22 16:07 dconeybe

Last night's tests had 5 failures: https://github.com/firebase/firebase-cpp-sdk/actions/runs/2628749371

4 of them failed due to TransactionTest.TestMaxAttempts, which is actively being investigated. https://github.com/firebase/firebase-cpp-sdk/pull/1012 will be merged today to improve logging to assist with tomorrow's investigation.

1 of them failed due to NumericTransformsTest.CreateDocumentWithIncrement timing out ("Waiting listener timed out") which is a known flake:

dconeybe avatar Jul 07 '22 15:07 dconeybe