gRPC abort in grpc::Channel::PerformOpsOnCall

[pschneider]

[REQUIRED] Step 2: Describe your environment

• Xcode version: 10.1
• Firebase SDK version: 5.13.0
• Firebase Component: Firestore
• Component version: Latest one included in 5.13.0 pod

[REQUIRED] Step 3: Describe the problem

We never saw this crash before. We thought it might be related to the recent switch from gRPC in #1968 and related ones.

The app is currently running in a small internal beta test with about 20 Users so far. The crash occurred on 3 different devices for 3 users from those 20. (iPhone 6, iPhone 6s, iPhone X. All are running iOS 12.0.1 or newer)

Steps to reproduce:

As far as we can see from our analytics screen views and the code we can just believe that the following state was in the app:

• We had one snapshot listener open which was about to being disposed (the user left the screen where this snapshot listener was running)
• It might have been that the document the snapshot listener was attached to was deleted or modified while the listener has been disposed

Stacktrace:

Crashed: com.google.firebase.firestore
SIGABRT ABORT 0x00000001d429b104

Crashed: com.google.firebase.firestore
0  libsystem_kernel.dylib         0x1d429b104 __pthread_kill + 8
1  libsystem_pthread.dylib        0x1d4316070 pthread_kill$VARIANT$mp + 380
2  libsystem_c.dylib              0x1d41f2d78 abort + 140
3  grpcpp                         0x10203d414 non-virtual thunk to grpc::Channel::PerformOpsOnCall(grpc::internal::CallOpSetInterface*, grpc::internal::Call*) (channel_cc.cc)
4  APP_NAME                        0x10139345c grpc::ClientAsyncReaderWriter<grpc::ByteBuffer, grpc::ByteBuffer>::Finish(grpc::Status*, void*) (call.h:680)
5  APP_NAME                        0x101393238 firebase::firestore::remote::GrpcStream::Shutdown() (functional:1862)
6  APP_NAME                        0x1013931c8 firebase::firestore::remote::GrpcStream::FinishImmediately() (grpc_stream.h:186)
7  APP_NAME                        0x1013935c0 firebase::firestore::remote::GrpcStream::WriteAndFinish(grpc::ByteBuffer&&) (optional.h:184)
8  APP_NAME                        0x1013a9704 firebase::firestore::remote::WriteStream::TearDown(firebase::firestore::remote::GrpcStream*) (byte_buffer.h:88)
9  APP_NAME                        0x1013a4a04 firebase::firestore::remote::Stream::Close(firebase::firestore::util::Status const&) (memory:2595)
10 APP_NAME                        0x1013a4868 firebase::firestore::remote::Stream::Stop() (memory:2595)
11 APP_NAME                        0x1012fb454 firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) (atomic:921)
12 APP_NAME                        0x1013032a8 firebase::firestore::util::internal::TimeSlot::InvokedByLibdispatch(void*) (executor_libdispatch.mm:173)
13 libdispatch.dylib              0x1d413e484 _dispatch_client_callout + 16
14 libdispatch.dylib              0x1d40e183c _dispatch_continuation_pop$VARIANT$mp + 412
15 libdispatch.dylib              0x1d40f1bac _dispatch_source_invoke$VARIANT$mp + 1704
16 libdispatch.dylib              0x1d40e5aac _dispatch_lane_serial_drain$VARIANT$mp + 284
17 libdispatch.dylib              0x1d40e6728 _dispatch_lane_invoke$VARIANT$mp + 432
18 libdispatch.dylib              0x1d40eeec8 _dispatch_workloop_worker_thread + 600
19 libsystem_pthread.dylib        0x1d43200dc _pthread_wqthread + 312
20 libsystem_pthread.dylib        0x1d4322cec start_wqthread + 4

com.google.firebase.firestore.rpc

com.google.firebase.firestore.rpc
0  libsystem_kernel.dylib         0x1d429d000 poll + 8
1  grpc                           0x101f39a3c pollset_work(grpc_pollset*, grpc_pollset_worker**, long long) (ev_poll_posix.cc:986)
2  grpc                           0x101f3c308 pollset_work(grpc_pollset*, grpc_pollset_worker**, long long) (ev_posix.cc:264)
3  grpc                           0x101f31270 cq_next(grpc_completion_queue*, gpr_timespec, void*) (completion_queue.cc:927)
4  grpcpp                         0x10203e968 grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) (completion_queue_cc.cc:57)
5  APP_NAME                        0x1012fdb50 firebase::firestore::remote::Datastore::PollGrpcQueue() (datastore.mm:128)
6  APP_NAME                        0x101303e6c firebase::firestore::util::internal::DispatchAsync(NSObject<OS_dispatch_queue>*, std::__1::function<void ()>&&)::$_0::__invoke(void*) (executor_libdispatch.mm:57)
7  libdispatch.dylib              0x1d413e484 _dispatch_client_callout + 16
8  libdispatch.dylib              0x1d40e5be0 _dispatch_lane_serial_drain$VARIANT$mp + 592
9  libdispatch.dylib              0x1d40e6728 _dispatch_lane_invoke$VARIANT$mp + 432
10 libdispatch.dylib              0x1d40eeec8 _dispatch_workloop_worker_thread + 600
11 libsystem_pthread.dylib        0x1d43200dc _pthread_wqthread + 312
12 libsystem_pthread.dylib        0x1d4322cec start_wqthread + 4

Thread #1

Thread
0  libsystem_kernel.dylib         0x1d429b374 __select_nocancel + 8
1  libsystem_dnssd.dylib          0x1d4232b78 deliver_request + 1000
2  libsystem_dnssd.dylib          0x1d4233ea0 DNSServiceCreateConnection + 116
3  libsystem_info.dylib           0x1d424c244 _mdns_search_ex + 940
4  libsystem_info.dylib           0x1d424bdac _mdns_search + 128
5  libsystem_info.dylib           0x1d424b2a8 mdns_addrinfo + 940
6  libsystem_info.dylib           0x1d4251118 search_addrinfo + 264
7  libsystem_info.dylib           0x1d42551c4 si_addrinfo + 1672
8  libsystem_info.dylib           0x1d4248964 _getaddrinfo_internal + 196
9  libsystem_info.dylib           0x1d4248894 getaddrinfo + 56
10 grpc                           0x101f6dba0 posix_blocking_resolve_address(char const*, char const*, grpc_resolved_addresses**) (resolve_address_posix.cc:87)
11 grpc                           0x101f6de98 do_request_thread(void*, grpc_error*) (resolve_address_posix.cc:157)
12 grpc                           0x101f3d2b0 GrpcExecutor::RunClosures(grpc_closure_list) (executor.cc:70)
13 grpc                           0x101f3d440 GrpcExecutor::ThreadMain(void*) (executor.cc:175)
14 grpc                           0x101f93338 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*)::'lambda'(void*)::__invoke(void*) (thd_posix.cc:101)
15 libsystem_pthread.dylib        0x1d431f2ac _pthread_body + 128
16 libsystem_pthread.dylib        0x1d431f20c _pthread_start + 48
17 libsystem_pthread.dylib        0x1d4322cf4 thread_start + 4

Thread #2


Thread
0  libsystem_kernel.dylib         0x1d429af0c __psynch_cvwait + 8
1  libsystem_pthread.dylib        0x1d4317cd8 _pthread_cond_wait$VARIANT$mp + 636
2  grpc                           0x101f8c950 gpr_cv_wait (sync_posix.cc:91)
3  grpc                           0x101f95e64 timer_thread(void*) (trace.h:72)
4  grpc                           0x101f93338 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*)::'lambda'(void*)::__invoke(void*) (thd_posix.cc:101)
5  libsystem_pthread.dylib        0x1d431f2ac _pthread_body + 128
6  libsystem_pthread.dylib        0x1d431f20c _pthread_start + 48
7  libsystem_pthread.dylib        0x1d4322cf4 thread_start + 4

Thread #3

Thread
0  libsystem_kernel.dylib         0x1d429af0c __psynch_cvwait + 8
1  libsystem_pthread.dylib        0x1d4317cd8 _pthread_cond_wait$VARIANT$mp + 636
2  grpc                           0x101f8c940 gpr_cv_wait (sync_posix.cc:89)
3  grpc                           0x101f95e64 timer_thread(void*) (trace.h:72)
4  grpc                           0x101f93338 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*)::'lambda'(void*)::__invoke(void*) (thd_posix.cc:101)
5  libsystem_pthread.dylib        0x1d431f2ac _pthread_body + 128
6  libsystem_pthread.dylib        0x1d431f20c _pthread_start + 48
7  libsystem_pthread.dylib        0x1d4322cf4 thread_start + 4

In the "Keys" entry of the crash log there's the following entry:

crash_info_entry_0
abort() called

These are all stack traces related to firestore / grpc i believe. There is no other information in the crash log.

I hope i could provide some information to maybe find the cause of this crash.

Kind regards

[ntnmrndn]

We have exactly the same issue. I guess we should downgrade firebase while waiting for a fix

[var-const]

Thank you for the report. This certainly looks related to the recent gRPC transition. I will try to reproduce this and will follow up here.

[var-const]

We have exactly the same issue.

@ntnmrndn Does your stack trace also look identical?

[ntnmrndn]

@var-const I have the same stack trace, and a second one in addition (almost identical).

See:

Crashed: com.google.firebase.firestore
0  libsystem_kernel.dylib                0x19ac5f104 __pthread_kill + 8
1  libsystem_pthread.dylib               0x19acde998 pthread_kill$VARIANT$armv81 + 296
2  libsystem_c.dylib                     0x19abb6d78 abort + 140
3  grpcpp                                0x1054d5020 grpc::Channel::PerformOpsOnCall(grpc::internal::CallOpSetInterface*, grpc::internal::Call*) + 622
4  Alpaca                                0x1031c272c grpc::ClientAsyncReaderWriter<grpc::ByteBuffer, grpc::ByteBuffer>::Finish(grpc::Status*, void*) (async_stream.h:570)
5  Alpaca                                0x1031c203c firebase::firestore::remote::GrpcStream::Shutdown() (functional:1862)
6  Alpaca                                0x1031c1eb8 firebase::firestore::remote::GrpcStream::FinishImmediately() (grpc_stream.h:186)
7  Alpaca                                0x1031c2bd8 firebase::firestore::remote::GrpcStream::WriteAndFinish(grpc::ByteBuffer&&) (optional.h:184)
8  Alpaca                                0x10321fd04 firebase::firestore::remote::WriteStream::TearDown(firebase::firestore::remote::GrpcStream*) (byte_buffer.h:88)
9  Alpaca                                0x10320a9d8 firebase::firestore::remote::Stream::Close(firebase::firestore::util::Status const&) (memory:2595)
10 Alpaca                                0x10320a3ec firebase::firestore::remote::Stream::Stop() (memory:2595)
11 Alpaca                                0x1030090f0 firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) (atomic:921)
12 Alpaca                                0x10302b97c firebase::firestore::util::internal::TimeSlot::Execute() (executor_libdispatch.mm:187)
13 Alpaca                                0x10302b7a8 firebase::firestore::util::internal::TimeSlot::InvokedByLibdispatch(void*) (executor_libdispatch.mm:173)
14 libclang_rt.asan_ios_dynamic.dylib    0x105af7030 asan_dispatch_call_block_and_release + 312
15 libdispatch.dylib                     0x19ab02484 _dispatch_client_callout + 16
16 libdispatch.dylib                     0x19aad8e14 _dispatch_continuation_pop$VARIANT$armv81 + 404
17 libdispatch.dylib                     0x19aae8ab4 _dispatch_source_invoke$VARIANT$armv81 + 1704
18 libdispatch.dylib                     0x19aadce84 _dispatch_lane_serial_drain$VARIANT$armv81 + 248
19 libdispatch.dylib                     0x19aaddaf4 _dispatch_lane_invoke$VARIANT$armv81 + 412
20 libdispatch.dylib                     0x19aae5f14 _dispatch_workloop_worker_thread + 584
21 libsystem_pthread.dylib               0x19ace40dc _pthread_wqthread + 312
22 libsystem_pthread.dylib               0x19ace6cec start_wqthread + 4

We also got a new crash in objc_object::release() but it might very well be on us :D

[pschneider]

We sadly could not reproduce this crash in our development environment yet but as soon as I have more information I'll provide it here.

[willtr101]

We got this same issue, seems related to the newly replaced gRPC:

[zackshapiro]

I'm also getting this as well.

I send a notification whenever the app backgrounds to any view controller listening to call .remove() on the appropriate listeners (for good measure and to conserve phone resources) since I don't need to keep track of anything in the background. And another notification to re-attach snapshot listeners for anything listening but neither of those events seem to be related to this.

I'm unable to recreate locally but I see about 1/3 of my test users experiencing this. Happens when they open the app. Otherwise I don't have any additional info

From my Podfile.lock

- FirebaseAuth (5.0.5):
- FirebaseAuthInterop (1.0.0)
- FirebaseCore (5.1.8):
- FirebaseFirestore (0.15.0):
- FirebaseFirestore/abseil-cpp (0.15.0):

[zackshapiro]

I'm downgrading to FirebaseFirestore 0.14.0 to see if it fixes the issue since #1968 was merged 20 days ago and 0.14.0 is the last build before. Will report back with results

After some testing, downgrading seems to have fixed the issue

[var-const]

Ignore my previous post; looks like that function can indeed be optimized away.

[willtr101]

UPDATED: We fixed the following issue by adding pod 'Firebase/Firestore', '~> 5.12.0'

@zackshapiro can you please share how to deal with the downgrade. We used

  pod 'FirebaseCore', '~> 5.1.7' 
  pod 'FirebaseFirestore', '~> 0.14.0'
  pod 'Firebase/Database'
  pod 'Firebase/Storage'
  pod 'Firebase/Messaging'
  pod 'Firebase/Performance'
  pod 'Firebase/RemoteConfig'
  pod 'Firebase/InAppMessagingDisplay'

  pod 'FirebaseUI/Firestore'

However, we faced this error:

  In Podfile:
    FirebaseUI/Firestore was resolved to 5.2.2, which depends on
      Firebase/Firestore

Specs satisfying the `Firebase/Firestore` dependency were found, but they required a higher minimum deployment target.
CocoaPods could not find compatible versions for pod "FirebaseFirestore":
  In Podfile:
    FirebaseFirestore (~> 0.14.0)

    FirebaseUI/Firestore was resolved to 5.2.2, which depends on
      Firebase/Firestore was resolved to 5.13.0, which depends on
        FirebaseFirestore (= 0.15.0)

Any ideas?

[var-const]

We found a bug which might be the cause of the issue (hard to say with full confidence, because the issue is so hard to reproduce, but the symptoms look similar). We're actively working on a fix, and I will post an update here as soon as I have it.

[zackshapiro]

@BusinessEngine I just bumped the version of FirebaseFirestore down from 0.15.0 to 0.14.0 and did a pod install. That's it :)

[pschneider]

@var-const Thank you very much for your work on this! Very appreciated!