yugabyte-db
yugabyte-db copied to clipboard
yugabyte
[DocDB][ASAN Unit Test Failure][2.8] YbAdminSnapshotScheduleTest.DeleteIndexOnRestore
Jira Link: DB-3441
Description
On 2.8.10.0-b2 YbAdminSnapshotScheduleTest.DeleteIndexOnRestore unit test is failing with a heap-use-after-free:
[m-1] =================================================================
[m-1] ==8636==ERROR: AddressSanitizer: heap-use-after-free on address 0x6130000b8960 at pc 0x7f953b3bb5f2 bp 0x7f95031c08d0 sp 0x7f95031c08c8
[m-1] READ of size 8 at 0x6130000b8960 thread T148 (raft [worker]xx)
[m-1] #0 0x7f953b3bb5f1 in std::__1::shared_ptr<std::__1::function<unsigned long ()> >::get() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/memory:3911:49
[m-1] #1 0x7f953b3bb5f1 in std::__1::shared_ptr<std::__1::function<unsigned long ()> >::operator bool() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/memory:3922
[m-1] #2 0x7f953b3bb5f1 in rocksdb::MutableCFOptions::MaxFileSizeForCompaction() const /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/util/mutable_cf_options.cc:75
[m-1] #3 0x7f953b11892b in rocksdb::VersionStorageInfo::CalculateBaseBytes(rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:1887:15
[m-1] #4 0x7f953b117db9 in rocksdb::Version::PrepareApply(rocksdb::MutableCFOptions const&, bool) /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:1036:17
[m-1] #5 0x7f953b128041 in rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*) /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:2310:10
[m-1] #6 0x7f953afc49d9 in rocksdb::DBImpl::ApplyVersionEdit(rocksdb::VersionEdit*) /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/db_impl.cc:5984:28
[m-1] #7 0x7f953afca7d7 in rocksdb::DBImpl::ModifyFlushedFrontier(yb::clone_ptr<rocksdb::UserFrontier>, rocksdb::FrontierModificationMode) /nfusr/centos-gcp-cloud/jenkins-worker-w2m61z/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-126/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/db_impl.cc:5996:10
Full log: https://gist.github.com/def-/a675079a98ade9099c0154718d30e3f7 https://drive.google.com/file/d/1jMyUeSgsYf5kn1MUQs-5KbNdkH0dbrVT/view?usp=sharing The stack trace looks similar to https://github.com/yugabyte/yugabyte-db/issues/13858 @druzac Should we consider this a duplicate?
Also happened in YbAdminSnapshotScheduleTest.ConsecutiveRestore:
[m-1] ==================
[m-1] WARNING: ThreadSanitizer: heap-use-after-free (pid=29081)
[m-1] Read of size 8 at 0x7b480019bce0 by thread T58 (mutexes: write M729718705982151408):
[m-1] #0 std::__1::shared_ptr<std::__1::function<unsigned long ()> >::get() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/tsan/libcxx/include/c++/v1/memory:3911:49 (librocksdb.so+0x4a037d)
[m-1] #1 std::__1::shared_ptr<std::__1::function<unsigned long ()> >::operator bool() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/tsan/libcxx/include/c++/v1/memory:3922 (librocksdb.so+0x4a037d)
[m-1] #2 rocksdb::MutableCFOptions::MaxFileSizeForCompaction() const /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/util/mutable_cf_options.cc:75 (librocksdb.so+0x4a037d)
[m-1] #3 rocksdb::VersionStorageInfo::CalculateBaseBytes(rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:1887:15 (librocksdb.so+0x3958ae)
[m-1] #4 rocksdb::Version::PrepareApply(rocksdb::MutableCFOptions const&, bool) /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:1036:17 (librocksdb.so+0x3954c2)
[m-1] #5 rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*) /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/version_set.cc:2310:10 (librocksdb.so+0x39acfb)
[m-1] #6 rocksdb::DBImpl::ApplyVersionEdit(rocksdb::VersionEdit*) /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/db_impl.cc:5984:28 (librocksdb.so+0x3148ba)
[m-1] #7 rocksdb::DBImpl::ModifyFlushedFrontier(yb::clone_ptr<rocksdb::UserFrontier>, rocksdb::FrontierModificationMode) /nfusr/centos-gcp-cloud/jenkins-worker-vfbr5n/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-tsan-459/yugabyte-db/build/tsan-clang7-dynamic-ninja/../../src/yb/rocksdb/db/db_impl.cc:5996:10 (librocksdb.so+0x316b50)
https://gist.github.com/def-/13d6b65ea02df104fbe790ef3b88f0a2 Seems like it can fail many unit tests
The most recent errors are related to restore.
Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-gcc11-fastdebug/1028/artifact/build/fastdebug-gcc11-dynamic-ninja/yb-test-logs/tests-tools__yb-admin-snapshot-schedule-test/YbAdminSnapshotScheduleTest_DeleteIndexOnRestore.log
../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:3182
Failed
Bad status: Runtime error (yb/util/subprocess.cc:612): Subprocess '/nfusr/alma8-gcp-cloud/jenkins-worker-tzqeu3/jenkins/jenkins-github-yugabyte-db-alma8-master-gcc11-fastdebug-1028/yugabyte-db/build/fastdebug-gcc11-dynamic-ninja/tests-tools/../bin/yb-admin' terminated with non-zero exit status 256: Error running restore_snapshot_schedule: Illegal state (yb/master/master_snapshot_coordinator.cc:672): Trying to restore to { days: 19286 time: 10:44:55.466393 } which is earlier than the configured retention. Not allowed. Earliest snapshot that can be used is cb35ff70-3045-426d-90d0-9ad8114f5f97 and was taken at { days: 19286 time: 10:44:49.218947 }.