Fleece crash during document retrieval

[philmitchell]

Describe the bug Couchbase Lite crashes sporadically in production. Crashes occur when retrieving multiple docs inside a loop, the docs are all the same type.

Logs

9   libc++abi.dylib                      0x0000000199762f58 std::__terminate(void (*)()) + 16
10  libc++abi.dylib                      0x0000000199762ef4 std::terminate() + 60
11  CouchbaseLiteSwift                   0x000000010112ea94 fleece::MDict<objc_object* __strong>::get(fleece::slice) const (Fleece.cc:253)
12  CouchbaseLiteSwift                   0x000000010112e9e8 _get(fleece::MDict<objc_object* __strong>&, NSString*) (CBLDictionary.mm:132)
13  CouchbaseLiteSwift                   0x000000010112eb24 _getObject(fleece::MDict<objc_object* __strong>&, NSString*, objc_class*) (CBLDictionary.mm:137)
14  CouchbaseLiteSwift                   0x000000010112c8b0 -[CBLDictionary valueForKey:] (CBLDictionary.mm:147)
15  CouchbaseLiteSwift                   0x0000000100fd64d8 CouchbaseLiteSwift.DictionaryObject.toDictionary() -> [Swift.String : Any] (DictionaryObject.swift:81)
16  CouchbaseLiteSwift                   0x0000000100fd6584 CouchbaseLiteSwift.DictionaryObject.toDictionary() -> [Swift.String : Any] (DictionaryObject.swift:215)
17  CouchbaseLiteSwift                   0x0000000100fde354 CouchbaseLiteSwift.ArrayObject.toArray() -> [Any] (ArrayObject.swift:192)
18  CouchbaseLiteSwift                   0x0000000100fd6690 CouchbaseLiteSwift.DictionaryObject.toDictionary() -> [Swift.String : Any] (DictionaryObject.swift:217)
19  CouchbaseLiteSwift                   0x0000000100fde354 CouchbaseLiteSwift.ArrayObject.toArray() -> [Any] (ArrayObject.swift:192)
20  CouchbaseLiteSwift                   0x0000000100fd6690 CouchbaseLiteSwift.DictionaryObject.toDictionary() -> [Swift.String : Any] (DictionaryObject.swift:217)
21  CouchbaseLiteSwift                   0x0000000100fde354 CouchbaseLiteSwift.ArrayObject.toArray() -> [Any] (ArrayObject.swift:192)
22  CouchbaseLiteSwift                   0x0000000100fe8ca4 CouchbaseLiteSwift.Document.toDictionary() -> [Swift.String : Any] (Document.swift:208)
23  MyApp                            0x000000010028c2ec MyApp.retrieveMyDocuments() -> [MyDocuments]

Platform:

• Device: iPhone 11 Pro Max, iPhone 7
• iOS: 15.2.1, 15.1
• CouchbaseLiteSwift 2.8.4

[philmitchell]

Closing this issue bc I suspect multithreading issue on my side.

[philmitchell]

Reopening bc current docs suggest that multithreading should not be a problem here.

[jayahariv]

Is it possible to share the whole crash report

[philmitchell]

Attaching lightly edited crash report - I have others that are very similar. crash_report-2517578-8e3bd82f.txt

[jayahariv]

If I want to try reproduce the same, could you please provide below information(preferably with similar code snippets you are using)

1. what steps would you recommend?
2. explain how you are using multiple threads?
3. A sample document with data. (seems like there is a lot nested dictionary > array > dictionary > array >.... )

[philmitchell]

1. what steps would you recommend? I'm not able to reproduce this crash. The two possible causes I can think of are:

   1. Multithreading (see below)
   2. Some "illegal" data that, although stored successfully, cbl is unable to retrieve.

2. explain how you are using multiple threads? The database always gets opened on the main thread. In the four crash reports I've examined, the crash occurs when accessing the db on a different thread.

3. A sample document with data. (seems like there is a lot nested dictionary > array > dictionary > array >.... ) Attached is a sample StudyHistorySet. Based on the sequence of cbl calls, I'm fairly confident that the sequence is: toDictionary(): Document is StudyHistorySet toArray(): Array of StudyHistory dicts toDictionary(): StudyHistory dict toArray(): Array of StudyHistory dicts toDictionary(): StudyHistory dict toArray(): Array of StudyHistory dicts toDictionary(): StudyHistory dict toDictionary(): TrialSeries dict

As you can see, it does contain nested dictionaries.

shs10360.json.txt

[philmitchell]

Also, the crash always occurs when calling getGroupHistorySets(forContentBundle:). The code is:

func getGroupHistorySets(forContentBundle contentBundle: ContentBundle) -> [StudyHistorySet]  {
    let contentGroups = contentBundle.contentGroups
    let historySets: [StudyHistorySet] = contentGroups.compactMap { fetchStorable(forDocumentType: .studyHistorySet(itemType: .group, itemId: $0.id)) }
    return historySets
}

func fetchStorable<T: Storable>(forDocumentType documentType: StorableDocumentType) -> T? {
    let id = documentType.id(withUserId: userId)
    guard let document = database.document(withID: id) else {
        return nil
    }
    return T.fromStorage(document.toDictionary())
}

[jayahariv]

Will be tracking this issue here CBL-3101

[jayahariv]

1. From the code snippet shared above, doesn't show any issue: (1) getting the document using the docID, and (2) document converted to dictionary.

The database always gets opened on the main thread. In the four crash reports I've examined, the crash occurs when accessing the db on a different thread.

If you access the database from different thread, it shouldn't have crashed. How are you accessing the db from different thread? Could you share some snippets? May be include the code snippets about the threads as well.

3. Nesting 6-7 levels also not an issue.

4. Also can you share the logic where the JSON is parsed and set into a CBLMutableDocument?

[philmitchell]

@jayahariv Thanks for not giving up on this, much appreciated!

First, let me mention that I've released a version of my app that does all its CBL work on a dedicated thread. I have still seen one crash after making this change.

In looking again at the five crash reports I have, I noticed two things:

1. This crash seems to always happen right after app launch For example (from crash report):

  Date/Time:       2021-12-01T16:48:04.999Z
  Launch Time:     2021-12-01T16:47:53Z

2. It seems to happen always during the same code sequence. That sequence starts with a method that does all its work on a background queue:

   let userInitiatedQueue = DispatchQueue(label: "contentBundlesViewModel.userInitiated", qos: .userInitiated)

   func theMethodThatCrashes(completion: (Result) -> ()) {
       userInitiatedQueue.async { [weak self] in
          ...
            crash here
          ...
       }
   }

Inside this method, the code path always leads to an init method of another class, and inside init there's a call that hits the db.

3. You asked for "the logic where the JSON is parsed and set into a CBLMutableDocument"

I guess you mean when saving docs to the db? It always goes through this method:

func createStorable<T: Storable>(from storable: T) {
    let id = storable.documentType.id(withUserId: userId)
    let storableDict = storable.toStorage()
    let document = MutableDocument(id: id, data: storableDict)
    document.setString(storable.documentType.name, forKey: Keys.documentType.rawValue)
    document.setString(userId, forKey: Keys.userId.rawValue)
    do {
        try database.saveDocument(document)            
    } catch {
        logger.error("Failed to create storable: \(error)")
    }
}

In the crash, the objects getting retrieved are always StudyHistorySets (which contain StudyHistory objects). This is how these objects get saved:

// Converting StudyHistorySet to dict
func toStorage() -> StorableDictionary {
    var stored: StorableDictionary = [
      .itemId : itemId,
      .itemType : itemType.rawValue,
    ]
    if historiesByIdSet.count > 0 {
        var historyKeys: [[Int]] = []
        var historyValues: [StorableDictionary] = []
        historiesByIdSet.forEach { entry in
            historyKeys.append(Array(entry.key))
            let history = entry.value
            let dict = history.toStorage()
            historyValues.append(dict)
        }
        stored[.historyKeys] = historyKeys
        stored[.historyValues] = historyValues
    }
    return stored
}

// Converting StudyHistory to dict
func toStorage() -> StorableDictionary {
    var stored: StorableDictionary = [
      .itemId : itemId,
      .itemType : itemType.rawValue,
      .isMarkedForRemedialStudy : _isMarkedForRemedialStudy,
      .isMarkedForEarlierReview : _isMarkedForEarlierReview,
      .isMarkedForDelayedReview : isMarkedForDelayedReview,
      .criterionTier : criterionTier.rawValue,
      .numberOfTrials : numberOfTrials,
      .numberOfTrialsSinceLastDueDateSet : numberOfTrialsSinceLastDueDateSet,
      .numberOfSessions : numberOfSessions,
      .numberOfErrors : numberOfErrors
    ]
    addOptional(toDict: &stored, key: .dateDue, value: $dateDue.getStringValue())
    addOptional(toDict: &stored, key: .trialSeries, value: trialSeries?.toStorage())
    addOptional(toDict: &stored, key: .sessionSeries, value: sessionSeries?.toStorage())
    let criterionScoresByTierDict = criterionScoresByTier?.mapEntries { entry in
        return (entry.key.stringValue, entry.value.value) // (criterionTier, clampedValue)
    }
    addOptional(toDict: &stored, key: .criterionScoresByTier, value: criterionScoresByTierDict)
    if let countOfConfusionsByLearnableId = self.countOfConfusionsByLearnableId {
        let (countOfConfusionsKeys, countOfConfusionsValues) = countOfConfusionsByLearnableId.parallelize()
        stored[.countOfConfusionsKeys] = countOfConfusionsKeys
        stored[.countOfConfusionsValues] = countOfConfusionsValues
    }
    if childHistoryMap.count > 0 {
        let values = childHistoryMap.values.map { $0.toStorage() }
        stored[.childHistoryMapValues] = values
    }
    return stored
}

[jayahariv]

I have a project attached, which does the similar steps mentioned, but not able to reproduce.

1. Save the shared JSON via main thread.
2. Read the document from userInitiated thread.
3. Convert the document to dictionary via doc->toDictionary()

I can rule out:

• data issue with the attached input JSON.
• save from main thread and different thread to call doc->toDictionary()

Also the above covertion logic(StudyHistorySet/StudyHistory to dict) seems like not depended on Couchbase Lite.

SampleProj.zip

[jayahariv]

@philmitchell I will be closing the tracking Jira ticket. We can keep this GH issue open for couple more days, and if you can gather more info which helps us track down the issue. please do post info, I can open a new Jira issue.