amplify-swift Initial sync may deadlock

Describe the bug

When running the initial sync, the app can deadlock. Interestingly enough, I'm currently only seeing this, when putting breakpoints in certain positions. I believe, however, that if it can be triggered with the debugger, it can also happen without.

Steps To Reproduce

Steps to reproduce the behavior:
1. Erase App
2. Place breakpoints in `ModelSyncedEventEmitter`
2.1. Line 119, `initialSyncOperationFinished = true`
2.2. Line 156, `sendModelSyncedEvent()`
2.3. Line 162, `sendModelSyncedEvent()`
3. Start App, triggering sync from scratch
4. As soon as debugger stops, click continue
5. App freezes completely.

Expected behavior

I expect the App to never freeze.

Amplify Framework Version

1.26.1

Amplify Categories

DataStore

Dependency manager

Cocoapods

Swift version

swift-driver version: 1.45.2 Apple Swift version 5.6 (swiftlang-5.6.0.323.62 clang-1316.0.20.8)

CLI version

8.2.0

Xcode version

Xcode 13.3.1, Build version 13E500a

Relevant log output

No response

Is this a regression?

No

Regression additional context

No response

Device

Simulator

iOS Version

15.4

Specific to simulators

unknown

Additional context

Here is where the threads are stuck: Main: Screenshot 2022-06-05 at 09 33 52

Amazon Threads: Screenshot 2022-06-05 at 09 35 52

Screenshot 2022-06-05 at 09 39 34

Jun 05 '22 07:06 anatomybook

Thanks for opening this @anatomybook. Could you please post the stack trace in text form? You can copy and paste it from the Debug Navigator where you took those screenshots. Thanks!

Jun 06 '22 11:06 atierian

@atierian Ok. Thanks for your answer! Here it is:

Thread 1 Queue : com.apple.main-thread (serial)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f711eb8 in gcd_queue_item_enqueue_hook ()
#3	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#4	0x000000010c591913 in dispatch_async ()
#5	0x000000010c7885fa in wrap_dispatch_async ()
#6	0x00007fff255f06f0 in __57-[_UIAfterCACommitQueue _enqueueCommitResponse:forPhase:]_block_invoke ()
#7	0x00007fff289cb474 in CA::Transaction::run_commit_handlers(CATransactionPhase) ()
#8	0x00007fff2899694e in CA::Context::commit_transaction(CA::Transaction*, double, double*) ()
#9	0x00007fff289cae86 in CA::Transaction::commit() ()
#10	0x00007fff2508d2db in _UIApplicationFlushRunLoopCATransactionIfTooLate ()
#11	0x00007fff251498bf in __processEventQueue ()
#12	0x00007fff2513fccd in __eventFetcherSourceCallback ()
#13	0x00007fff20373833 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ ()
#14	0x00007fff2037372b in __CFRunLoopDoSource0 ()
#15	0x00007fff20372bf8 in __CFRunLoopDoSources0 ()
#16	0x00007fff2036d2f4 in __CFRunLoopRun ()
#17	0x00007fff2036ca90 in CFRunLoopRunSpecific ()
#18	0x00007fff2cb72c8e in GSEventRunModal ()
#19	0x00007fff2508e90e in -[UIApplication _run] ()
#20	0x00007fff25093569 in UIApplicationMain ()
#21	0x00007fff5d9ce17d in closure #1 in KitRendererCommon(_:) ()
#22	0x00007fff5d9ce0b7 in runApp<τ_0_0>(_:) ()
#23	0x00007fff5d390c60 in static App.main() ()
#24	0x000000010aa5d593 in static WhatsLeftApp.$main() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/WhatsLeft/WhatsLeftApp.swift:11
#25	0x000000010aa5ddc9 in main ()
#26	0x000000010c3baf21 in start_sim ()
#27	0x000000011362951e in start ()
com.apple.uikit.eventfetch-thread (6)#0	0x00007fff6faaa97a in mach_msg_trap ()
#1	0x00007fff6faaace8 in mach_msg ()
#2	0x00007fff20372e58 in __CFRunLoopServiceMachPort ()
#3	0x00007fff2036d46e in __CFRunLoopRun ()
#4	0x00007fff2036ca90 in CFRunLoopRunSpecific ()
#5	0x00007fff20827e31 in -[NSRunLoop(NSRunLoop) runMode:beforeDate:] ()
#6	0x00007fff208280aa in -[NSRunLoop(NSRunLoop) runUntilDate:] ()
#7	0x00007fff25151d1e in -[UIEventFetcher threadMain] ()
#8	0x00007fff20850f89 in __NSThread__start__ ()
#9	0x00007fff6fb094e1 in _pthread_start ()
#10	0x00007fff6fb04f6b in thread_start ()
com.apple.NSURLConnectionLoader (10)#0	0x00007fff6faaa97a in mach_msg_trap ()
#1	0x00007fff6faaace8 in mach_msg ()
#2	0x00007fff20372e58 in __CFRunLoopServiceMachPort ()
#3	0x00007fff2036d46e in __CFRunLoopRun ()
#4	0x00007fff2036ca90 in CFRunLoopRunSpecific ()
#5	0x00007fff24066684 in ___lldb_unnamed_symbol11158$$CFNetwork ()
#6	0x00007fff20850f89 in __NSThread__start__ ()
#7	0x00007fff6fb094e1 in _pthread_start ()
#8	0x00007fff6fb04f6b in thread_start ()
Thread 48 Queue : com.amazonaws.ModelSyncedEventEmitterQueue (serial)
#0	0x00007fff6faae9e2 in __psynch_rw_wrlock ()
#1	0x00007fff6fb07f04 in _pthread_rwlock_lock_wait ()
#2	0x00007fff6fb06514 in _pthread_rwlock_lock_slow ()
#3	0x00007fff6f7155b2 in enter_stack_in_backtrace_uniquing_table ()
#4	0x00007fff6f7120f7 in gcd_queue_item_enqueue_hook ()
#5	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#6	0x000000010c591913 in dispatch_async ()
#7	0x000000010c7885fa in wrap_dispatch_async ()
#8	0x00007fff5967b917 in _swift_dispatch_async ()
#9	0x00007fff5967a836 in OS_dispatch_queue.async(group:qos:flags:execute:) ()
#10	0x00007fff59680c6e in OS_dispatch_queue.schedule(options:_:) ()
#11	0x00007fff59680f90 in protocol witness for Scheduler.schedule(options:_:) in conformance OS_dispatch_queue ()
#12	0x00007fff4ed4a294 in Publishers.ReceiveOn.Inner.receive(_:) ()
#13	0x00007fff4ed4a940 in protocol witness for Subscriber.receive(_:) in conformance Publishers.ReceiveOn<τ_0_0, τ_0_1>.Inner<τ_1_0> ()
#14	0x00007fff4ed76f7f in partial apply for closure #2 in Publishers._Merged.receive(_:_:) ()
#15	0x00007fff4ed77009 in partial apply for closure #1 in Publishers._Merged.receive(_:_:) ()
#16	0x00007fff4ed6af93 in Publishers._Merged.guardedApplyDownstream<τ_0_0>(_:) ()
#17	0x00007fff4ed6b737 in Publishers._Merged.receive(_:_:) ()
#18	0x00007fff4ed6b664 in Publishers._Merged.Side.receive(_:) ()
#19	0x00007fff4ed1ef84 in PassthroughSubject.Conduit.offer(_:) ()
#20	0x00007fff4ed20247 in partial apply for closure #1 in PassthroughSubject.send(_:) ()
#21	0x00007fff4ed57424 in ConduitList.forEach(_:) ()
#22	0x00007fff4ed1ea25 in PassthroughSubject.send(_:) ()
#23	0x0000000111e28996 in ModelSyncedEventEmitter.sendModelSyncedEvent() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/InitialSync/ModelSyncedEventEmitter.swift:178
#24	0x0000000111e27a7b in ModelSyncedEventEmitter.onReceiveReconciliationEvent(value:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/InitialSync/ModelSyncedEventEmitter.swift:162
#25	0x0000000111e26379 in closure #4 in ModelSyncedEventEmitter.init(modelSchema:initialSyncOrchestrator:reconciliationQueue:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/InitialSync/ModelSyncedEventEmitter.swift:82
#26	0x00007fff4ed12de3 in Subscribers.Sink.receive(_:) ()
#27	0x00007fff4ed134a0 in protocol witness for Subscriber.receive(_:) in conformance Subscribers.Sink<τ_0_0, τ_0_1> ()
#28	0x00007fff4ed4a3af in closure #1 in Publishers.ReceiveOn.Inner.receive(_:) ()
#29	0x00007fff59680cc9 in thunk for @escaping @callee_guaranteed () -> () ()
#30	0x000000010c7886db in __wrap_dispatch_async_block_invoke ()
#31	0x000000010c58c816 in _dispatch_call_block_and_release ()
#32	0x000000010c58da5b in _dispatch_client_callout ()
#33	0x000000010c5943bd in _dispatch_lane_serial_drain ()
#34	0x000000010c5950ed in _dispatch_lane_invoke ()
#35	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#36	0x00007fff6fb05fd0 in _pthread_wqthread ()
#37	0x00007fff6fb04f57 in start_wqthread ()
Thread 58 Queue : com.amazonaws.RemoteSyncEngineOperationQueue (serial)
#0	0x00007fff6faaa9b6 in semaphore_wait_trap ()
#1	0x000000010c58dfea in _dispatch_sema4_wait ()
#2	0x000000010c58e4e6 in _dispatch_semaphore_wait_slow ()
#3	0x0000000111f15a7e in RemoteSyncEngine.performInitialSync() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/RemoteSyncEngine.swift:353
#4	0x0000000111f0c575 in RemoteSyncEngine.respond(to:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/RemoteSyncEngine.swift:173
#5	0x0000000111f0ba0d in closure #1 in closure #1 in RemoteSyncEngine.init(storageAdapter:dataStoreConfiguration:authModeStrategy:outgoingMutationQueue:mutationEventIngester:mutationEventPublisher:initialSyncOrchestratorFactory:reconciliationQueueFactory:stateMachine:networkReachabilityPublisher:requestRetryablePolicy:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/RemoteSyncEngine.swift:149
#6	0x0000000111a2fa71 in thunk for @escaping @callee_guaranteed () -> () ()
#7	0x000000010c7886db in __wrap_dispatch_async_block_invoke ()
#8	0x000000010c58c816 in _dispatch_call_block_and_release ()
#9	0x000000010c58da5b in _dispatch_client_callout ()
#10	0x000000010c5943bd in _dispatch_lane_serial_drain ()
#11	0x000000010c5950ed in _dispatch_lane_invoke ()
#12	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#13	0x00007fff6fb05fd0 in _pthread_wqthread ()
#14	0x00007fff6fb04f57 in start_wqthread ()
Thread 64 Queue : NSOperationQueue 0x615000070880 (QOS: UNSPECIFIED) (concurrent)
#0	0x00007fff6faaa9b6 in semaphore_wait_trap ()
#1	0x000000010c58dfea in _dispatch_sema4_wait ()
#2	0x000000010c58e4e6 in _dispatch_semaphore_wait_slow ()
#3	0x000000010d36a20c in AWSAuthService.getToken() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSPluginsCore/AmplifyPlugins/Core/AWSPluginsCore/Auth/AWSAuthService.swift:148
#4	0x000000010d36e5f8 in protocol witness for AWSAuthServiceBehavior.getToken() in conformance AWSAuthService ()
#5	0x000000010d35647b in BasicUserPoolTokenProvider.getToken() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSPluginsCore/AmplifyPlugins/Core/AWSPluginsCore/Auth/Provider/AuthTokenProvider.swift:24
#6	0x000000010d35665e in protocol witness for AuthTokenProvider.getToken() in conformance BasicUserPoolTokenProvider ()
#7	0x0000000111af4ced in AuthTokenURLRequestInterceptor.intercept(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/API/AWSAPICategoryPlugin/Interceptor/RequestInterceptor/AuthTokenURLRequestInterceptor.swift:33
#8	0x0000000111af5bcd in protocol witness for URLRequestInterceptor.intercept(_:) in conformance AuthTokenURLRequestInterceptor ()
#9	0x0000000111c0df3d in closure #1 in AWSGraphQLOperation.main() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/API/AWSAPICategoryPlugin/Operation/AWSGraphQLOperation.swift:108
#10	0x0000000111c0e6dd in partial apply for closure #1 in AWSGraphQLOperation.main() ()
#11	0x00007fff30abc0de in Sequence.reduce<τ_0_0>(_:_:) ()
#12	0x0000000111c0aa70 in AWSGraphQLOperation.main() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/API/AWSAPICategoryPlugin/Operation/AWSGraphQLOperation.swift:106
#13	0x0000000111c0ef88 in @objc AWSGraphQLOperation.main() ()
#14	0x00000001102d8be3 in AsynchronousOperation.start() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/Amplify/Amplify/Core/Support/AsychronousOperation.swift:71
#15	0x00000001102d8ca8 in @objc AsynchronousOperation.start() ()
#16	0x00007fff207f119d in __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ ()
#17	0x00007fff207f0ce4 in __NSOQSchedule_f ()
#18	0x000000010c58da5b in _dispatch_client_callout ()
#19	0x000000010c59093b in _dispatch_block_invoke_direct ()
#20	0x000000010c7886db in __wrap_dispatch_async_block_invoke ()
#21	0x000000010c58c816 in _dispatch_call_block_and_release ()
#22	0x000000010c58da5b in _dispatch_client_callout ()
#23	0x000000010c59034e in _dispatch_continuation_pop ()
#24	0x000000010c58f739 in _dispatch_async_redirect_invoke ()
#25	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#26	0x000000010c5a064e in _dispatch_worker_thread2 ()
#27	0x00007fff6fb05f8a in _pthread_wqthread ()
#28	0x00007fff6fb04f57 in start_wqthread ()
Thread 66#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f711eb8 in gcd_queue_item_enqueue_hook ()
#3	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#4	0x000000010c5a6a88 in _dispatch_mach_reply_merge_msg ()
#5	0x000000010c5aeb74 in _dispatch_event_loop_merge ()
#6	0x000000010c5a145a in _dispatch_workloop_worker_thread ()
#7	0x00007fff6fb05fd0 in _pthread_wqthread ()
#8	0x00007fff6fb04f57 in start_wqthread ()
Thread 69 Queue : com.apple.network.connections (serial)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5902ca in _dispatch_continuation_pop ()
#5	0x000000010c5a4fe1 in _dispatch_source_invoke ()
#6	0x000000010c596739 in _dispatch_workloop_invoke ()
#7	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#8	0x00007fff6fb05fd0 in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 71#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f711eb8 in gcd_queue_item_enqueue_hook ()
#3	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#4	0x000000010c591913 in dispatch_async ()
#5	0x000000010c7885fa in wrap_dispatch_async ()
#6	0x00007fff23ff71a3 in ___lldb_unnamed_symbol8452$$CFNetwork ()
#7	0x000000010df93f16 in -[AWSURLSessionManager invalidate] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Networking/AWSURLSessionManager.m:239
#8	0x000000010dea3ec2 in -[AWSNetworking dealloc] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Networking/AWSNetworking.m:101
#9	0x00007fff201a27d7 in objc_object::sidetable_release(bool, bool) ()
#10	0x000000010d6aec1b in -[AWSCognitoIdentityProvider .cxx_destruct] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCognitoIdentityProvider/AWSCognitoIdentityProvider/AWSCognitoIdentityProviderService.m:169
#11	0x00007fff20188ed2 in object_cxxDestructFromClass(objc_object*, objc_class*) ()
#12	0x00007fff2019be07 in objc_destructInstance ()
#13	0x00007fff201a1f69 in -[NSObject dealloc] ()
#14	0x00007fff201a27d7 in objc_object::sidetable_release(bool, bool) ()
#15	0x000000010d72abe7 in -[AWSCognitoIdentityUserPool .cxx_destruct] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCognitoIdentityProvider/AWSCognitoIdentityProvider/AWSCognitoIdentityUserPool.m:35
#16	0x00007fff20188ed2 in object_cxxDestructFromClass(objc_object*, objc_class*) ()
#17	0x00007fff2019be07 in objc_destructInstance ()
#18	0x00007fff201a1f69 in -[NSObject dealloc] ()
#19	0x000000010d721583 in -[AWSCognitoIdentityUserPool dealloc] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCognitoIdentityProvider/AWSCognitoIdentityProvider/AWSCognitoIdentityUserPool.m:197
#20	0x00007fff201a27d7 in objc_object::sidetable_release(bool, bool) ()
#21	0x000000010e2cc044 in __swift_destroy_boxed_opaque_existential_1 ()
#22	0x000000010e2f8c81 in @objc FetchUserPoolTokensOperation.__ivar_destroyer ()
#23	0x00007fff20188ed2 in object_cxxDestructFromClass(objc_object*, objc_class*) ()
#24	0x00007fff2019be07 in objc_destructInstance ()
#25	0x00007fff201a1f69 in -[NSObject dealloc] ()
#26	0x00007fff207eb562 in -[NSOperation dealloc] ()
#27	0x00007fff201a27d7 in objc_object::sidetable_release(bool, bool) ()
#28	0x00007fff207b930c in NSKVOPendingNotificationRelease ()
#29	0x00007fff202f4b81 in __CFArrayReleaseValues ()
#30	0x00007fff20374e86 in _CFRelease ()
#31	0x00007fff204431b4 in __CFBasicHashDrain ()
#32	0x00007fff20374e86 in _CFRelease ()
#33	0x00007fff207b9525 in NSKeyValueObservingTSDDestroy ()
#34	0x00007fff203605bc in __CFTSDFinalize ()
#35	0x00007fff6fb07567 in _pthread_tsd_cleanup ()
#36	0x00007fff6fb09b89 in _pthread_exit ()
#37	0x00007fff6fb07310 in _pthread_wqthread_exit ()
#38	0x00007fff6fb06062 in _pthread_wqthread ()
#39	0x00007fff6fb04f57 in start_wqthread ()
Thread 72 Queue : com.apple.root.user-initiated-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5902ca in _dispatch_continuation_pop ()
#5	0x000000010c5a4fe1 in _dispatch_source_invoke ()
#6	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#7	0x00007fff6fb05fd0 in _pthread_wqthread ()
#8	0x00007fff6fb04f57 in start_wqthread ()
Thread 73 Queue : com.amazonaws.SyncEventEmitter (serial)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712528 in gcd_queue_item_complete_hook ()
#3	0x000000010c5943e3 in _dispatch_lane_serial_drain ()
#4	0x000000010c5950ed in _dispatch_lane_invoke ()
#5	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#6	0x00007fff6fb05fd0 in _pthread_wqthread ()
#7	0x00007fff6fb04f57 in start_wqthread ()
Thread 74 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 75 Queue : com.amazonaws.ReconcileAndLocalSaveOperation (serial)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f711eb8 in gcd_queue_item_enqueue_hook ()
#3	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#4	0x000000010c591913 in dispatch_async ()
#5	0x000000010c7885fa in wrap_dispatch_async ()
#6	0x00007fff5967b917 in _swift_dispatch_async ()
#7	0x00007fff5967a836 in OS_dispatch_queue.async(group:qos:flags:execute:) ()
#8	0x00007fff59680c6e in OS_dispatch_queue.schedule(options:_:) ()
#9	0x00007fff59680f90 in protocol witness for Scheduler.schedule(options:_:) in conformance OS_dispatch_queue ()
#10	0x00007fff4ed4a294 in Publishers.ReceiveOn.Inner.receive(_:) ()
#11	0x00007fff4ed4a940 in protocol witness for Subscriber.receive(_:) in conformance Publishers.ReceiveOn<τ_0_0, τ_0_1>.Inner<τ_1_0> ()
#12	0x00007fff4ed1ef84 in PassthroughSubject.Conduit.offer(_:) ()
#13	0x00007fff4ed20247 in partial apply for closure #1 in PassthroughSubject.send(_:) ()
#14	0x00007fff4ed57a24 in partial apply for thunk for @callee_guaranteed (@guaranteed ConduitBase<τ_0_0, τ_0_1>) -> (@error @owned Error) ()
#15	0x00007fff30a035cd in Sequence.forEach(_:) ()
#16	0x00007fff4ed5740d in ConduitList.forEach(_:) ()
#17	0x00007fff4ed1ea25 in PassthroughSubject.send(_:) ()
#18	0x0000000111c34cf7 in AWSIncomingEventReconciliationQueue.onReceiveValue(receiveValue:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/AWSIncomingEventReconciliationQueue.swift:123
#19	0x0000000111c3416d in implicit closure #8 in implicit closure #7 in AWSIncomingEventReconciliationQueue.init(modelSchemas:api:storageAdapter:syncExpressions:auth:authModeStrategy:modelReconciliationQueueFactory:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/AWSIncomingEventReconciliationQueue.swift:84
#20	0x00007fff4ed12de3 in Subscribers.Sink.receive(_:) ()
#21	0x00007fff4ed134a0 in protocol witness for Subscriber.receive(_:) in conformance Subscribers.Sink<τ_0_0, τ_0_1> ()
#22	0x00007fff4ed1ef84 in PassthroughSubject.Conduit.offer(_:) ()
#23	0x00007fff4ed20247 in partial apply for closure #1 in PassthroughSubject.send(_:) ()
#24	0x00007fff4ed57424 in ConduitList.forEach(_:) ()
#25	0x00007fff4ed1ea25 in PassthroughSubject.send(_:) ()
#26	0x0000000111c583ac in closure #2 in AWSModelReconciliationQueue.enqueue(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/AWSModelReconciliationQueue.swift:170
#27	0x00007fff4ed12de3 in Subscribers.Sink.receive(_:) ()
#28	0x00007fff4ed134a0 in protocol witness for Subscriber.receive(_:) in conformance Subscribers.Sink<τ_0_0, τ_0_1> ()
#29	0x00007fff4ed1ef84 in PassthroughSubject.Conduit.offer(_:) ()
#30	0x00007fff4ed20247 in partial apply for closure #1 in PassthroughSubject.send(_:) ()
#31	0x00007fff4ed57a24 in partial apply for thunk for @callee_guaranteed (@guaranteed ConduitBase<τ_0_0, τ_0_1>) -> (@error @owned Error) ()
#32	0x00007fff30a035cd in Sequence.forEach(_:) ()
#33	0x00007fff4ed5740d in ConduitList.forEach(_:) ()
#34	0x00007fff4ed1ea25 in PassthroughSubject.send(_:) ()
#35	0x0000000111ee595e in ReconcileAndLocalSaveOperation.notify(savedModel:mutationType:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:446
#36	0x0000000111ee3d3f in closure #1 in closure #1 in ReconcileAndLocalSaveOperation.saveMetadata(storageAdapter:applyResult:mutationType:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:413
#37	0x0000000111fd3edc in closure #1 in SQLiteStorageEngineAdapter.save<τ_0_0>(_:modelSchema:condition:completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Storage/SQLite/StorageEngineAdapter+SQLite.swift:182
#38	0x0000000111fd4c79 in SQLiteStorageEngineAdapter.query<τ_0_0>(_:modelSchema:predicate:sort:paginationInput:completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Storage/SQLite/StorageEngineAdapter+SQLite.swift:278
#39	0x0000000111fd0e2e in SQLiteStorageEngineAdapter.save<τ_0_0>(_:modelSchema:condition:completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Storage/SQLite/StorageEngineAdapter+SQLite.swift:178
#40	0x0000000111fce6b4 in SQLiteStorageEngineAdapter.save<τ_0_0>(_:condition:completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Storage/SQLite/StorageEngineAdapter+SQLite.swift:130
#41	0x0000000111fe210e in protocol witness for ModelStorageBehavior.save<τ_0_0>(_:condition:completion:) in conformance SQLiteStorageEngineAdapter ()
#42	0x0000000111ee328d in closure #1 in ReconcileAndLocalSaveOperation.saveMetadata(storageAdapter:applyResult:mutationType:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:406
#43	0x0000000111eec536 in partial apply for closure #1 in ReconcileAndLocalSaveOperation.saveMetadata(storageAdapter:applyResult:mutationType:) ()
#44	0x00007fff4ed37161 in specialized Future.init(_:) ()
#45	0x00007fff4ed347a0 in Future.__allocating_init(_:) ()
#46	0x0000000111ee2305 in ReconcileAndLocalSaveOperation.saveMetadata(storageAdapter:applyResult:mutationType:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:400
#47	0x0000000111edceec in closure #1 in closure #1 in closure #1 in ReconcileAndLocalSaveOperation.applyRemoteModelsDispositions(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:293
#48	0x0000000111eed8a5 in partial apply for closure #1 in closure #1 in closure #1 in ReconcileAndLocalSaveOperation.applyRemoteModelsDispositions(_:) ()
#49	0x00007fff4edc7ee3 in Publishers.FlatMap.Outer.receive(_:) ()
#50	0x00007fff4edc97f0 in protocol witness for Subscriber.receive(_:) in conformance Publishers.FlatMap<τ_0_0, τ_0_1>.Outer<τ_1_0> ()
#51	0x00007fff4ed36077 in Future.Conduit.request(_:) ()
#52	0x00007fff4ed57922 in protocol witness for Subscription.request(_:) in conformance ConduitBase<τ_0_0, τ_0_1> ()
#53	0x00007fff4edc7e4c in Publishers.FlatMap.Outer.receive(subscription:) ()
#54	0x00007fff4edc97d0 in protocol witness for Subscriber.receive(subscription:) in conformance Publishers.FlatMap<τ_0_0, τ_0_1>.Outer<τ_1_0> ()
#55	0x00007fff4ed359e0 in Future.Conduit.subscribe() ()
#56	0x00007fff4ed3509f in Future.receive<τ_0_0>(subscriber:) ()
#57	0x00007fff4ed36ab0 in protocol witness for Publisher.receive<τ_0_0>(subscriber:) in conformance Future<τ_0_0, τ_0_1> ()
#58	0x00007fff4edc6b2e in Publishers.FlatMap.receive<τ_0_0>(subscriber:) ()
#59	0x00007fff4ed6a988 in Publishers.MergeMany.receive<τ_0_0>(subscriber:) ()
#60	0x00007fff4ed42a9d in Publishers.Collect.receive<τ_0_0>(subscriber:) ()
#61	0x00007fff4ed126e8 in Publisher.sink(receiveCompletion:receiveValue:) ()
#62	0x0000000111edab98 in closure #1 in ReconcileAndLocalSaveOperation.applyRemoteModelsDispositions(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:321
#63	0x00007fff4ed37161 in specialized Future.init(_:) ()
#64	0x00007fff4ed347a0 in Future.__allocating_init(_:) ()
#65	0x0000000111ed9b32 in ReconcileAndLocalSaveOperation.applyRemoteModelsDispositions(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:262
#66	0x0000000111ed349d in closure #2 in closure #2 in ReconcileAndLocalSaveOperation.reconcile(remoteModels:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AmplifyPlugins/AmplifyPlugins/DataStore/AWSDataStoreCategoryPlugin/Sync/SubscriptionSync/ReconcileAndLocalSave/ReconcileAndLocalSaveOperation.swift:139
#67	0x0000000111ed35d0 in thunk for @escaping @callee_guaranteed (@guaranteed [MutationSync<AnyModel>], @guaranteed [MutationSyncMetadata]) -> (@owned Future<(), DataStoreError>) ()
#68	0x0000000111eeef41 in partial apply for thunk for @escaping @callee_guaranteed (@guaranteed [MutationSync<AnyModel>], @guaranteed [MutationSyncMetadata]) -> (@owned Future<(), DataStoreError>) ()
#69	0x00007fff4edc7ee3 in Publishers.FlatMap.Outer.receive(_:) ()
#70	0x00007fff4edc97f0 in protocol witness for Subscriber.receive(_:) in conformance Publishers.FlatMap<τ_0_0, τ_0_1>.Outer<τ_1_0> ()
#71	0x00007fff4edc700b in Publishers.FlatMap.Outer.receiveInner(_:_:) ()
#72	0x00007fff4edc6f14 in Publishers.FlatMap.Outer.Side.receive(_:) ()
#73	0x00007fff4ed36077 in Future.Conduit.request(_:) ()
#74	0x00007fff4ed57922 in protocol witness for Subscription.request(_:) in conformance ConduitBase<τ_0_0, τ_0_1> ()
#75	0x00007fff4edc6eec in Publishers.FlatMap.Outer.receiveInner(subscription:_:) ()
#76	0x00007fff4edc6e04 in Publishers.FlatMap.Outer.Side.receive(subscription:) ()
#77	0x00007fff4ed359e0 in Future.Conduit.subscribe() ()
#78	0x00007fff4ed3509f in Future.receive<τ_0_0>(subscriber:) ()
#79	0x00007fff4ed36ab0 in protocol witness for Publisher.receive<τ_0_0>(subscriber:) in conformance Future<τ_0_0, τ_0_1> ()
#80	0x00007fff4edc7fe9 in Publishers.FlatMap.Outer.receive(_:) ()
#81	0x00007fff4edc97f0 in protocol witness for Subscriber.receive(_:) in conformance Publishers.FlatMap<τ_0_0, τ_0_1>.Outer<τ_1_0> ()
#82	0x00007fff4ed02984 in Publishers.SubscribeOn.Inner.receive(_:) ()
#83	0x00007fff4ed02ac0 in protocol witness for Subscriber.receive(_:) in conformance Publishers.SubscribeOn<τ_0_0, τ_0_1>.Inner<τ_1_0> ()
#84	0x00007fff4ed36077 in Future.Conduit.request(_:) ()
#85	0x00007fff4ed57922 in protocol witness for Subscription.request(_:) in conformance ConduitBase<τ_0_0, τ_0_1> ()
#86	0x00007fff4ed02458 in closure #1 in Publishers.SubscribeOn.Inner.request(_:) ()
#87	0x00007fff4ed03c17 in partial apply for closure #1 in Publishers.SubscribeOn.Inner.request(_:) ()
#88	0x00007fff59680cc9 in thunk for @escaping @callee_guaranteed () -> () ()
#89	0x000000010c7886db in __wrap_dispatch_async_block_invoke ()
#90	0x000000010c58c816 in _dispatch_call_block_and_release ()
#91	0x000000010c58da5b in _dispatch_client_callout ()
#92	0x000000010c5943bd in _dispatch_lane_serial_drain ()
#93	0x000000010c5950ed in _dispatch_lane_invoke ()
#94	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#95	0x00007fff6fb05fd0 in _pthread_wqthread ()
#96	0x00007fff6fb04f57 in start_wqthread ()
Thread 76 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 77 Queue : com.amazonaws.ModelSyncedEventEmitterQueue (serial)
#0	0x00007fff6faae9e2 in __psynch_rw_wrlock ()
#1	0x00007fff6fb07f04 in _pthread_rwlock_lock_wait ()
#2	0x00007fff6fb06514 in _pthread_rwlock_lock_slow ()
#3	0x00007fff6f715957 in remove_stack_from_backtrace_uniquing_table ()
#4	0x00007fff6f712c4f in decrement_work_item_refcount ()
#5	0x00007fff6f712530 in gcd_queue_item_complete_hook ()
#6	0x000000010c5943e3 in _dispatch_lane_serial_drain ()
#7	0x000000010c5950ed in _dispatch_lane_invoke ()
#8	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#9	0x00007fff6fb05fd0 in _pthread_wqthread ()
#10	0x00007fff6fb04f57 in start_wqthread ()
Thread 79 Queue : com.amazonaws.ModelSyncedEventEmitterQueue (serial)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712528 in gcd_queue_item_complete_hook ()
#3	0x000000010c5943e3 in _dispatch_lane_serial_drain ()
#4	0x000000010c5950ed in _dispatch_lane_invoke ()
#5	0x000000010c5a1601 in _dispatch_workloop_worker_thread ()
#6	0x00007fff6fb05fd0 in _pthread_wqthread ()
#7	0x00007fff6fb04f57 in start_wqthread ()
Thread 80 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f711eb8 in gcd_queue_item_enqueue_hook ()
#3	0x000000010c5c004a in _dispatch_introspection_queue_item_enqueue_hook ()
#4	0x000000010c591913 in dispatch_async ()
#5	0x000000010c7885fa in wrap_dispatch_async ()
#6	0x00007fff5967b917 in _swift_dispatch_async ()
#7	0x00007fff5967a836 in OS_dispatch_queue.async(group:qos:flags:execute:) ()
#8	0x000000010e2f0a97 in FetchUserPoolTokensOperation.acceptEvent(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/AWSMobileClientOperations/AWSAsyncOperations/GetTokensUserPool/FetchUserPoolTokensOperation.swift:57
#9	0x000000010e2f5519 in closure #1 in FetchUserPoolTokensOperation.fetchToken() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/AWSMobileClientOperations/AWSAsyncOperations/GetTokensUserPool/FetchUserPoolTokensOperation.swift:142
#10	0x000000010e2db601 in closure #1 in AWSCognitoIdentityUser.getUserPoolToken(completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/Helpers/UserPoolHelper/CognitoIdentityUserBehavior.swift:28
#11	0x000000010e2db8e1 in partial apply for closure #1 in AWSCognitoIdentityUser.getUserPoolToken(completion:) ()
#12	0x000000010e276eab in thunk for @escaping @callee_guaranteed (@guaranteed AWSTask<AWSCognitoIdentityUserSession>) -> (@out Any?) ()
#13	0x000000010df3e74b in __56-[AWSTask continueWithExecutor:block:cancellationToken:]_block_invoke at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Bolts/AWSTask.m:331
#14	0x000000010de18bf0 in __30+[AWSExecutor defaultExecutor]_block_invoke_2 at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Bolts/AWSExecutor.m:65
#15	0x000000010de1b246 in -[AWSExecutor execute:] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Bolts/AWSExecutor.m:131
#16	0x000000010df3e17b in -[AWSTask continueWithExecutor:block:cancellationToken:] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Bolts/AWSTask.m:368
#17	0x000000010df3fa6f in -[AWSTask continueWithBlock:] at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSCore/AWSCore/Bolts/AWSTask.m:375
#18	0x000000010e2db006 in AWSCognitoIdentityUser.getUserPoolToken(completion:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/Helpers/UserPoolHelper/CognitoIdentityUserBehavior.swift:26
#19	0x000000010e2db98e in protocol witness for CognitoIdentityUserBehavior.getUserPoolToken(completion:) in conformance AWSCognitoIdentityUser ()
#20	0x000000010e2f4455 in FetchUserPoolTokensOperation.fetchToken() at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/AWSMobileClientOperations/AWSAsyncOperations/GetTokensUserPool/FetchUserPoolTokensOperation.swift:129
#21	0x000000010e2f2d62 in closure #1 in closure #1 in FetchUserPoolTokensOperation.acceptEvent(_:) at /Users/ben/Documents/20210826_WhatsLeft/WhatsLeft/Pods/AWSMobileClient/AWSAuthSDK/Sources/AWSMobileClient/AWSMobileClientOperations/AWSAsyncOperations/GetTokensUserPool/FetchUserPoolTokensOperation.swift:71
#22	0x000000010e27da71 in thunk for @escaping @callee_guaranteed () -> () ()
#23	0x000000010c7886db in __wrap_dispatch_async_block_invoke ()
#24	0x000000010c58c816 in _dispatch_call_block_and_release ()
#25	0x000000010c58da5b in _dispatch_client_callout ()
#26	0x000000010c58fe63 in _dispatch_queue_override_invoke ()
#27	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#28	0x000000010c5a064e in _dispatch_worker_thread2 ()
#29	0x00007fff6fb05f8a in _pthread_wqthread ()
#30	0x00007fff6fb04f57 in start_wqthread ()
Thread 81 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 82 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 83 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()
Thread 84 Queue : com.apple.root.default-qos (concurrent)
#0	0x00007fff6faac0ea in __ulock_wait ()
#1	0x00007fff6fafa27e in _os_unfair_lock_lock_slow ()
#2	0x00007fff6f712384 in gcd_queue_item_dequeue_hook ()
#3	0x000000010c5c0253 in _dispatch_introspection_queue_item_dequeue_hook ()
#4	0x000000010c5a4bfa in _dispatch_source_invoke ()
#5	0x000000010c58fc94 in _dispatch_queue_override_invoke ()
#6	0x000000010c59fba2 in _dispatch_root_queue_drain ()
#7	0x000000010c5a064e in _dispatch_worker_thread2 ()
#8	0x00007fff6fb05f8a in _pthread_wqthread ()
#9	0x00007fff6fb04f57 in start_wqthread ()

Jun 06 '22 13:06 anatomybook

I'd like to add, that it takes precisely the same dataset, that is needed to trigger https://github.com/aws-amplify/amplify-ios/issues/1864. So somehow this seems to be related.

Jun 06 '22 13:06 anatomybook

Something I just noticed and to avoid confusion when you follow the trace above: In ModelSyncedEventEmitter.swift, there's an offset of 6 lines, since my commented hack was in there. For completeness, it looked like this when it ran:

Screenshot 2022-06-06 at 15 43 53

Otherwise the files are identical to your version, except for those commented lines. As I've pointed out in https://github.com/aws-amplify/amplify-ios/issues/1864, this hack is needed for the ready signal to trigger.

Jun 06 '22 13:06 anatomybook

Thanks for this thorough report @anatomybook, it's very helpful. We'll update here with any news.

Jun 15 '22 16:06 atierian

Hi @anatomybook, really appreciate the level of details you've provided and your patience. Regarding the change in https://github.com/aws-amplify/amplify-ios/issues/1864 to make sure ModelSyncedEvent is emitted, it look like the correct change. I cut the PR for that issue in https://github.com/aws-amplify/amplify-ios/pull/1989 under the branch fix/modelsynced-sync-finished. Since that PR fix has your code change, we can continue to debug the deadlock issue you're seeing with this PR branch (while it's in PR), to avoid confusion.

The PR adds your code change, then as a first step towards investigating the deadlock issue, i've moved the queue off of the global queue in the second commit (https://github.com/aws-amplify/amplify-ios/pull/1989/commits/96804bca6aae377ff8f59be00322437a57a2d74a). I'm not 100% if this is the problem but I recall discussing this topic with @atierian briefly about stop using the global queue so we wanted to clean up the code around this anyways.

Glancing over the code, the intention of the private queue in ModelSyncedEventEmitter is to serialize the asynchronous work of processing both the sync operation events and reconciliation events. There are even variables like dispatchedModelSyncedEvent that is just a bool that is made thread safe because of the access to it is made only under the work put on the queue.

Do you have a schema (redacted if needed) and expected number of models persisted that you can provide us to replicate a similar environment to reproduce the issue? Could you try and reproduce the deadlock again using fix/modelsynced-sync-finished branch?

Jul 08 '22 21:07 lawmicha

Hello @lawmicha ,

sorry for my delayed reply. Here's my schema - slightly edited:

type ActorZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  .
  .
  .
  myContents: [ContentsZ] @connection(name: "LinkToActor")
  groups: [GroupZ] @connection(name: "LinkToGroups")
  myLists: [ListZ] @connection(name: "LinkToLists")
}

type ContentsZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  creation: AWSDateTime!
  dateOnReceipt: AWSDateTime!
  .
  .
  .
  actor: ActorZ @connection(name: "LinkToActor")
  contents: [ContentsFieldZ] @connection(name: "LinkToContent")
  filenames: [ImageZ] @connection(name: "LinkToImages")
}

type ImageZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  sThreeKey: String!
  lastLookedAt: AWSDateTime
  remarks: String
  details: String
  receipt: ContentsZ @connection(name: "LinkToImages")
}

type ContentsFieldZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  caption: String!
  orderByMe: Int!
  price: Int
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  parent: ContentsZ @connection(name: "LinkToContent")
  groups: [ContentsFieldVsGroupZ] @connection(keyName: "byContentsField", fields: ["id"])
}

type ContentsFieldVsGroupZ @model(queries: null) @auth(rules: [{ allow: owner }])
@key(name: "byContentsField", fields: ["myContentsFieldId", "groupId"])
@key(name: "byGroupA", fields: ["groupId", "myContentsFieldId"])
{
  id: ID!
  myContentsFieldId: ID!
  groupId: ID!
  myContentsField: ContentsFieldZ! @connection(fields: ["myContentsFieldId"])
  group: GroupZ! @connection(fields: ["groupId"])
}

type GroupZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  actor: ActorZ @connection(name: "LinkToGroups")
  searches: [SearchZ] @connection(name: "LinkToSearches")
  prognosis: [PrognosisZ] @connection(name: "LinkToPrognosis")
  statusChangeTime: [StatusChangeTimeZ] @connection(name: "LinkToInCartTime")
  myLists: [ListVsGroupZ] @connection(keyName: "byGroup", fields: ["id"])
  myContentsFields: [ContentsFieldVsGroupZ] @connection(keyName: "byGroupA", fields: ["id"])
}

type SearchZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  phrase: String
  comment: String
  .
  .
  .
  .
  .
  .
  threshold: Float
  group: GroupZ @connection(name: "LinkToSearches")
}

type StatusChangeTimeZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  .
  statusChangeTime: AWSDateTime!
  group: GroupZ @connection(name: "LinkToInCartTime")
}

type PrognosisZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  expected: Float!
  lower: Float!
  upper: Float!
  updated: AWSDateTime!
  .
  group: GroupZ @connection(name: "LinkToPrognosis")
}

type ListVsGroupZ @model(queries: null) @auth(rules: [{ allow: owner }])
@key(name: "byList", fields: ["myListId", "groupId"])
@key(name: "byGroup", fields: ["groupId", "myListId"])
{
  id: ID!
  amount: Float
  .
  .
  .
  .
  .
  .
  .
  myListId: ID!
  groupId: ID!
  myList: ListZ! @connection(fields: ["myListId"])
  group: GroupZ! @connection(fields: ["groupId"])
}

type ListZ @model @auth(rules: [{ allow: owner }])
{
  id: ID!
  name: String!
  thresholdPercentage: Float
  imageData: String
  imageDetails: String
  usageDetails: String
  .
  .
  .
  .
  creation: AWSDateTime!
  actor: ActorZ @connection(name: "LinkToLists")
  groups: [ListVsGroupZ] @connection(keyName: "byList", fields: ["id"])
}

The number of items is quite low in this user-account: 1 x ActorZ 6 x StatusChangeTimeZ 5 x GroupZ 8 x ContentsFieldVsGroupZ 102 x ContentsFieldZ 2 x ContentsZ 2 x ImageZ 5 x SearchZ 4 x ListVsGroupZ 1 x ListZ 0 x PrognosisZ

I hope this helps. I'll check out the deadlock tomorrow.

Thanks for your support!

Aug 03 '22 22:08 anatomybook

Could you try and reproduce the deadlock again using fix/modelsynced-sync-finished branch?

I just tried. Everything works - no more deadlock! It'd be awesome if that made it into main.

Thank you every much.

Aug 05 '22 19:08 anatomybook

Hi @anatomybook thanks for your patience, fix/modelsynced-sync-finished changes were merged in another PR that made it to main and just released in 1.28.0 https://github.com/aws-amplify/amplify-ios/releases/tag/v1.28.0

Keeping this issue open to look into the schema you've provided

Aug 08 '22 20:08 lawmicha

Keeping this issue open to look into the schema you've provided

@lawmicha - thanks for the inclusion of my suggestion.

Please let me know if you have any questions or need more details regarding the schema. Talking about - the schema is old. I haven't touched it in ages. Given that I started early during the development of DataStore, I'm sometimes wondering whether the issues I'm seeing are related to it being old and never updated. In case you have any hints - perhaps new and special keywords - that I may incorporate, please let me know. I haven't found anything, but perhaps I'm overlooking things.

Two headaches this schema still gives me: I'm currently still struggling with performance, which I've tracked down to this issue: https://github.com/aws-amplify/amplify-ios/issues/1579. Additionally, I'm seeing orphaned children quite often. https://github.com/aws-amplify/amplify-ios/issues/2059.

Aug 08 '22 21:08 anatomybook

amplify-swift amplify-swift copied to clipboard

Initial sync may deadlock

Describe the bug

Steps To Reproduce

Expected behavior

Amplify Framework Version

Amplify Categories

Dependency manager

Swift version

CLI version

Xcode version

Relevant log output

Is this a regression?

Regression additional context

Device

iOS Version

Specific to simulators

Additional context

amplify-swift
amplify-swift copied to clipboard