Crash in grpc_library.h line 42 under iOS and iPadOS 14

[JanGorman]

Step 1: Describe your environment

• Xcode version: 13.2.1
• Firebase SDK version: 8.13.0
• Installation method: CocoaPods
• Firebase Component: Firestore
• Target platform(s): iOS

Step 2: Describe the problem

Steps to reproduce:

Haven't been able to reproduce myself yet. The crash is spiking in our latest app release and only affects devices running iOS and iPadOS 14.

I've tried reproducing on a physical device running iOS 14 with Address Sanitizer and Undefined Behaviour Sanitizer enabled, alas without success.

The logs we get from Crashlytics:

Crashed: com.google.firebase.firestore EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000000

Crashed: com.google.firebase.firestore
0  FirebaseFirestore              0x2d514 grpc::GrpcLibraryCodegen::GrpcLibraryCodegen(bool) + 42 (grpc_library.h:42)
1  FirebaseFirestore              0x2d3d8 grpc_impl::CompletionQueue::CompletionQueue(grpc_completion_queue_attributes const&) + 247 (completion_queue_impl.h:247)
2  FirebaseFirestore              0x2b100 firebase::firestore::remote::Datastore::Datastore(firebase::firestore::core::DatabaseInfo const&, std::__1::shared_ptr<firebase::firestore::util::AsyncQueue> const&, std::__1::shared_ptr<firebase::firestore::credentials::CredentialsProvider<firebase::firestore::credentials::AuthToken, firebase::firestore::credentials::User> >, std::__1::shared_ptr<firebase::firestore::credentials::CredentialsProvider<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, firebase::firestore::remote::ConnectivityMonitor*, firebase::firestore::remote::FirebaseMetadataProvider*) + 104 (datastore.cc:104)
3  FirebaseFirestore              0x61750 firebase::firestore::core::FirestoreClient::Initialize(firebase::firestore::credentials::User const&, firebase::firestore::api::Settings const&) + 3237 (memory:3237)
4  FirebaseFirestore              0x6f4c firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) + 993 (atomic:993)
5  FirebaseFirestore              0x12aefc firebase::firestore::util::Task::ExecuteAndRelease() + 1871 (functional:1871)
6  libdispatch.dylib              0x61280 _dispatch_client_callout + 16
7  libdispatch.dylib              0x9dcc _dispatch_lane_serial_drain$VARIANT$mp + 612
8  libdispatch.dylib              0xa8a8 _dispatch_lane_invoke$VARIANT$mp + 424
9  libdispatch.dylib              0x14338 _dispatch_workloop_worker_thread + 712
10 libsystem_pthread.dylib        0xa5a4 _pthread_wqthread + 272
11 libsystem_pthread.dylib        0xd874 start_wqthread + 8

[schmidt-sebastian]

@JanGorman Thanks for reporting this. This looks like an issue in the underlying library and likely the same as https://github.com/grpc/grpc/issues/24850. I will ping someone from the GRPC team to gain some traction.

[EliotAndres]

I work with Jan, adding more information on this. Here's how the crashes look like:

We didn't update Firebase before the release started crashing, and we're a bit curious as to why it started with this release. Would you have any pointers on where to look?

[ondrejkorol]

We experienced the exact same issue! All affected devices were running iOS 14.8.1. grpc::GrpcLibraryCodegen::GrpcLibraryCodegen(bool) + 42

[schmidt-sebastian]

@ondrejkorol That's very interesting. Do you see any crashes with iOS 14.8.1 and prior versions of the Firestore SDK (< 8.13)?

[codingrhythm]

We don't have crashes from Firebase SDK, but we have similar crashes from the gRPC library we integrated with our App. The crash only happens for some iOS 14 users. We didn't bump gRPC between releases, this is what our stack trace looks like.

We even tried to release a previous build (which is not crashing) with just a version bump, the strange thing is the build with only a version bump started crashing as well with the same stack trace. This leads me to think, there might be something changed from the Apple side when recompiling the app from the bitcode.

Xcode 13.3 was released on 14th Mar 2022. The timeline aligns, but I am not sure if it could be the cause. When I compare the install size of 2 builds with only version difference. The second one has 1mb less.

Old version released before 14th Mar	New version released after 14 Mar with only version bump

The gRPC lib is integrated with the iOS app as an XCFramework via SPM. The iOS project is built using Xcode 13.2.1 as well.

[ondrejkorol]

@ondrejkorol That's very interesting. Do you see any crashes with iOS 14.8.1 and prior versions of the Firestore SDK (< 8.13)?

We have the prior version – FirebaseFirestore (8.12.1). Could the new version (8.13) solve it?

[codingrhythm]

Xcode 13.3.1 is out, the issue might have been solved if a new build is submitted.

When you target iOS 13.4 and later and enable bitcode, the bitcode segment is now stripped correctly. This fixes a crash that prevented some apps from running in older operating systems (iOS 14 and earlier). (90024354) (FB9951126)

[codingrhythm]

I am able to confirm that releasing another build without any changes except for a version bump resolves the issue,.

[JanGorman]

Thanks @codingrhythm for the detailed investigation and your follow-up. We submitted a new version as well with Xcode 13.3.1 that is still in review. I'll update here once it's released

[JanGorman]

Confirmed on our end as well that releasing with 13.3.1 fixes the issue. Thanks everyone

[porchgeneration]

We have had the exact same issue and now released a new build built with Xcode 13.3.1 and Firestore version 9 but we are still seeing the crash. ( before grpc_library.h line 43 now grpc_library.h line 45) Anyone else ?

[codingrhythm]

Same for us. We did further investigation and found out the crash is from the users who installed the app via MDM. The same build doesn't crash if it's installed via AppStore. The work around for us at the moment is to build the app with Xcode 13.2.1 with Bitcode disabled.

[ondrejkorol]

Yeah, it's back..

grpc_library.h - Line 42
grpc::GrpcLibraryCodegen::GrpcLibraryCodegen(bool) + 42

[porchgeneration]

Any news ? Are any of you getting the crash using Remote Config with Firebase ?

[dconeybe]

I'll take a look into this. The flaky nature of this issue suggest some memory mismanagement that sometimes you get away with, and other times you don't. I'll pour through Firestore's code and see what I can find. Any additional stack traces would be greatly appreciated.

[ondrejkorol]

Is there any progress please? iOS 14 is still crashing from time to time.

Crashed: com.google.firebase.firestore – EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000000
0  FirebaseFirestore              0x2f614 GrpcLibraryCodegen + 42 (grpc_library.h:42)
1  FirebaseFirestore              0x2f4d8 CompletionQueue + 247 (completion_queue_impl.h:247)
2  FirebaseFirestore              0x2d1c4 Datastore + 104 (datastore.cc:104)
3  FirebaseFirestore              0x637d0 Initialize + 958 (shared_ptr.h:958)
4  FirebaseFirestore              0x90e8 ExecuteBlocking + 997 (atomic:997)
5  FirebaseFirestore              0x119e58 ExecuteAndRelease + 491 (function.h:491)

[dconeybe]

@ondrejkorol Unfortunately, no progress yet. I have been stepping through the code line by line to try to figure out what might be causing this but I don't see anything. Could you tell me which version of the Firebase iOS SDK you are using? Also, is it possible that this crash is occurring when the app is shutting down?

[ondrejkorol]

@dconeybe We have Firebase Firestore v8.8. Looks like this is happening during the app launch when we're setting up a Firestore listener (observing a collection). The newest crashes are again tied to iOS 14.8.1.

@porchgeneration, @JanGorman are you still experiencing these crashes?

[dconeybe]

To anyone experiencing this bug, please upgrade to at least v8.15.0 of the Firebase iOS SDK (preferably the latest, which is v9.0.0 at the time of writing). Please report back if this upgrade fixes the crash. If it does not fix the crash, then please provide updated stack traces.

Please do not report crashes against older versions of the Firebase iOS SDK. Namely, if you see completion_queue_impl.h in your stack trace, then you are using an older version of the Firebase iOS SDK, as that file was deleted from grpc.

The reason for this request is that the Firebase iOS SDK upgraded its grpc dependency from 1.28.0 to 1.44.0 in Firebase iOS SDK v8.15.0. This upgrade may even fix the crash.

[JanGorman]

@ondrejkorol We're not seeing this crash. We've since updated to v8.15.0 (and moved to SPM) so as @dconeybe suggests, this will probably help in your case.

[paulb777]

Going to close, since it looks like the issue is addressed with the gRPC 1.44.0 available with Firebase 8.15.0+

[eroscai]

@dconeybe @paulb777 we are still receiving this crash, gRPC version and Firebase version have been updated to required version. The only difference is that XCode has been updated to the latest version 13.4.

environment:

• Xcode version: 13.4
• Firebase SDK version: 8.15.0
• Installation method: CocoaPods
• Firebase Component: Firestore
• Target platform(s): iOS

stack trace:

Crashed: com.google.firebase.firestore
0  FirebaseFirestore              0x2ed90 grpc::GrpcLibraryCodegen::GrpcLibraryCodegen(bool) + 45 (grpc_library.h:45)
1  FirebaseFirestore              0x2ec54 grpc::CompletionQueue::CompletionQueue(grpc_completion_queue_attributes const&) + 253 (completion_queue.h:253)
2  FirebaseFirestore              0x2c954 firebase::firestore::remote::Datastore::Datastore(firebase::firestore::core::DatabaseInfo const&, std::__1::shared_ptr<firebase::firestore::util::AsyncQueue> const&, std::__1::shared_ptr<firebase::firestore::credentials::CredentialsProvider<firebase::firestore::credentials::AuthToken, firebase::firestore::credentials::User> >, std::__1::shared_ptr<firebase::firestore::credentials::CredentialsProvider<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, firebase::firestore::remote::ConnectivityMonitor*, firebase::firestore::remote::FirebaseMetadataProvider*) + 104 (datastore.cc:104)
3  FirebaseFirestore              0x63204 firebase::firestore::core::FirestoreClient::Initialize(firebase::firestore::credentials::User const&, firebase::firestore::api::Settings const&) + 958 (shared_ptr.h:958)
4  FirebaseFirestore              0x844c firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) + 997 (atomic:997)
5  FirebaseFirestore              0x133414 firebase::firestore::util::Task::ExecuteAndRelease() + 491 (function.h:491)
6  libdispatch.dylib              0x481c _dispatch_client_callout + 20
7  libdispatch.dylib              0xc004 _dispatch_lane_serial_drain + 620
8  libdispatch.dylib              0xcc00 _dispatch_lane_invoke + 404
9  libdispatch.dylib              0x174bc _dispatch_workloop_worker_thread + 764
10 libsystem_pthread.dylib        0x37a4 _pthread_wqthread + 276
11 libsystem_pthread.dylib        0xa74c start_wqthread + 8

[paulb777]

@eroscai Please share the Podfile.lock

[eroscai]

@paulb777 Done!

Podfile.lock.zip

[dconeybe]

@eroscai are you able to reproduce the crash? If so, there is some extra logging I'd like you to add to help debug this.

[eroscai]

@dconeybe I can't reproduce this crash myself. It only happens on iOS 14, and it happens very beginning of the launch, because i can't see any logs from the Logs column of Crashlytics dashboard. We will try to use XCode 13.3.1 to archive and release the new version, if there is any progress, i will update here.

[eroscai]

By the way, I found some warnings during the build time, I don't know if it is related to this crash.

[dconeybe]

@eroscai Thanks for the update. I don't think those warnings are related to this crash. I still am completely stumped. Everything in the code looks just fine to me.

Here is my brain dump though:

The crash seems to be coming from the failed assertion at https://github.com/grpc/grpc/blob/93e689448564585c8ea8b8135481c2dc41b737bb/include/grpcpp/impl/codegen/grpc_library.h#L45-L47

GPR_CODEGEN_ASSERT(g_glip &&
                   "gRPC library not initialized. See "
                   "grpc::internal::GrpcLibraryInitializer.");

This means that g_glip must have been null.

Now g_glip is initialized here at https://github.com/grpc/grpc/blob/93e689448564585c8ea8b8135481c2dc41b737bb/include/grpcpp/impl/grpc_library.h#L39-L45

class GrpcLibraryInitializer final {
 public:
  GrpcLibraryInitializer() {
    if (grpc::g_glip == nullptr) {
      static auto* const g_gli = new GrpcLibrary();
      grpc::g_glip = g_gli;
    }

This GrpcLibraryInitializer is used as a global variable all over the place. This makes sure that g_glip gets initialized before main() is even invoked. Here are some examples: https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/src/cpp/server/server_credentials.cc#L24, https://github.com/grpc/grpc/blob/0f9d024fec6a96cfa07ebae633e3ee96c933d3c4/src/cpp/util/byte_buffer_cc.cc#L33, and https://github.com/grpc/grpc/blob/0f9d024fec6a96cfa07ebae633e3ee96c933d3c4/src/cpp/client/client_context.cc#L58.

I'll log an issue in the grpc respository and see if they have any insights.

[dconeybe]

https://github.com/grpc/grpc/issues/29843 opened

[codingrhythm]

Just install the app via MDM on an iOS 14 device to reproduce the crash.