Crash in `CBLDatabase_SaveDocumentWithConcurrencyControl` and `CBLDocument::properties()`

[blaugold]

These crashes were reported in https://github.com/cbl-dart/cbl-dart/issues/390:

Crashed: Thread :  SIGSEGV  0x0000000000000030
#00 pc 0x1ed108 libcblite.so 
litecore::Rev::body() const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RevTree.cc:178
#01 pc 0x1fb61c libcblite.so 
litecore::RevTreeRecord::currentRevBody() const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RevTreeRecord.cc:110
#02 pc 0x1fb61c libcblite.so 
litecore::RevTreeRecord::currentRevBody() const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RevTreeRecord.cc:110
#03 pc 0x18e62c libcblite.so 
litecore::TreeDocument::getRevisionBody() const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/Database/TreeDocument.cc:139
#04 pc 0x81d3c libcblite.so 
CBLDocument::properties() const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/src/CBLDocument_Internal.hh:121
#05 pc 0x8d9a0 libcblite.so
#06 pc 0x5aec18 libapp.so 
... 

Crashed: Thread :  SIGILL  0x0000007da4a01318
#00 pc 0xef318 libcblite.so 
void* fleece::offsetby<void>(void*, long)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/vendor/fleece/API/fleece/slice.hh:79
#01 pc 0x31a980 libcblite.so 
litecore::RawRevision::copyFrom(litecore::Rev const&)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RawRevTree.cc:138
#02 pc 0x31a5a8 libcblite.so 
litecore::RawRevision::encodeTree(std::__ndk1::vector<litecore::Rev*, std::__ndk1::allocator<litecore::Rev*> > const&, std::__ndk1::unordered_map<unsigned int, litecore::Rev const*, std::__ndk1::hash<unsigned int>, std::__ndk1::equal_to<unsigned int>, std::__ndk1::allocator<std::__ndk1::pair<unsigned int const, litecore::Rev const*> > > const&)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RawRevTree.cc:102
#03 pc 0x1ed444 libcblite.so 
litecore::RevTree::encode()
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RevTree.cc:94
#04 pc 0x1fc53c libcblite.so 
litecore::RevTreeRecord::save(litecore::ExclusiveTransaction&)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/RevTrees/RevTreeRecord.cc:214
#05 pc 0x18f5b8 libcblite.so 
litecore::TreeDocument::save(unsigned int)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/Database/TreeDocument.cc:343
#06 pc 0x192080 libcblite.so 
litecore::TreeDocument::saveNewRev(C4DocPutRequest const&, litecore::Rev const*, bool)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/Database/TreeDocument.cc:628
#07 pc 0x19043c libcblite.so 
litecore::TreeDocument::putNewRevision(C4DocPutRequest const&, C4Error*)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/Database/TreeDocument.cc:604
#08 pc 0x12417c libcblite.so 
C4Document::update(fleece::slice, unsigned char) const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/C/c4Document.cc:214
#09 pc 0x86a6c libcblite.so 
CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)::$_0::operator()(C4Database*) const
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/src/CBLDocument.cc:116
#10 pc 0x86764 libcblite.so 
void litecore::access_lock<fleece::Retained<C4Database>, std::__ndk1::recursive_mutex>::useLocked<CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)::$_0>(CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)::$_0)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/vendor/couchbase-lite-core/LiteCore/Support/access_lock.hh:160
#11 pc 0x813a0 libcblite.so 
void CBLDatabase::useLocked<CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)::$_0>(CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)::$_0)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/src/CBLDatabase_Internal.hh:284
#12 pc 0x810fc libcblite.so 
CBLDocument::save(CBLDatabase*, CBLDocument::SaveOptions const&)
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/src/CBLDocument.cc:77
#13 pc 0x72b84 libcblite.so 
CBLDatabase_SaveDocumentWithConcurrencyControl
/home/couchbase/jenkins/workspace/couchbase-lite-c-android/build_android_arm64/../couchbase-lite-c/src/CBLDatabase_CAPI.cc:194
#14 pc 0x5b3678 libapp.so
...

Both seem to be related to RevTree but that might be a coincidence.

[pasin]

Is it possible that the document that is accessing its properties or is being saved is released on another thread?

[blaugold]

Only a single thread is accessing the same database and all related documents. However, I do share the FLDict that CBLDocument_Properties returns with another thread. I'm never mutating a document's properties, though. Instead, before saving a document, I replace its properties with a newly built FLMutableDict.

[borrrden]

SIGILL is concerning to me. It's not something that is easy to come across unless you are hand writing assembly. If I had to guess, it would be the somehow the compiler is emitting extension instructions that are not supported on that particular device for whatever reason but I have no idea how to diagnose that or what to do about it at the moment. The other one is a simple null dereference that will need to be looked into.

[blaugold]

This issue was caused by https://github.com/couchbase/couchbase-lite-C/issues/339, which has been fixed in the latest release. In our case we write to the same document many times. In some cases more times than can be counted by a uint16_t variable. I suspect that there is a counter somewhere in the document revision data, that once overflowed causes corruption of the document data that could lead to crashes.