doris icon indicating copy to clipboard operation
doris copied to clipboard

doris多次down机

Open yanjinyin opened this issue 1 year ago • 1 comments

问题描述:drois运行一段时间会down机,麻烦 帮看下了 doris控制台,查询报错:

Execution failed: Error Failed to execute sql: java.sql.SQLException: (conn=4590) errCode = 2, detailMessage = 1593504 have no queryable replicas. err: 1593505's backend 17290 does not exist or not alive

BE报错信息:

OpenJDK 64-Bit Server VM warning: You have loaded library /data/doris/apache-doris-2.0.3-bin-x64/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. *** Query id: 9dd3e1af44b4ba8-a3e7867d300cc1f3 *** *** tablet id: 0 *** *** Aborted at 1708658397 (unix time) try "date -d @1708658397" if you are using GNU date *** *** Current BE git commitID: https://github.com/apache/doris/commit/37d31a5550aa6fb40ba5b6696385c7dcd3dbc931 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 113089 (TID 114635 OR 0x7f6e3988a700) from PID 0; stack trace: *** F0223 11:19:57.602829 113117 threadpool.cpp:247] Check failed: 1 == _tokens.size() (1 vs. 5) Threadpool SendBatchThreadPool destroyed with 5 allocated tokens *** Check failure stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417 1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 4# 0x00007F717D858400 in /lib64/libc.so.6

yanjinyin avatar Feb 23 '24 03:02 yanjinyin

*** Query id: 7bb6fdd5fbbb42f1-9d6b4f423d89fa13 *** *** tablet id: 0 *** *** Aborted at 1708667124 (unix time) try "date -d @1708667124" if you are using GNU date *** *** Current BE git commitID: 37d31a5 *** *** SIGSEGV unknown detail explain (@0x0) received by PID 127271 (TID 127694 OR 0x7f7debfd3700) from PID 0; stack trace: *** F0223 13:45:24.435146 127300 threadpool.cpp:247] Check failed: 1 == _tokens.size() (1 vs. 5) Threadpool SendBatchThreadPool destroyed with 5 allocated tokens *** Check failure stack trace: *** @ 0x561260630219 google::LogMessageFatal::~LogMessageFatal() @ 0x561258eb0a06 doris::ThreadPool::~ThreadPool() @ 0x561258c47742 doris::ExecEnv::~ExecEnv() @ 0x7f7ef0997ce9 __run_exit_handlers @ 0x7f7ef0997d37 __GI_exit @ 0x7f7eeb1fb7ac vm_direct_exit() @ 0x7f7eeb6c3287 VM_Operation::evaluate() @ 0x7f7eeb6c128c VMThread::evaluate_operation() @ 0x7f7eeb6c1709 VMThread::loop() @ 0x7f7eeb6c1bc2 VMThread::run() @ 0x7f7eeb49f382 java_start() @ 0x7f7ef002dea5 start_thread @ 0x7f7ef0a5c96d __clone @ (nil) (unknown) 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417 1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/lib/amd64/server/libjvm.so 4# 0x00007F7EF0994400 in /lib64/libc.so.6 5# doris::get_key_coder(doris::FieldType) at /root/src/doris-2.0/be/src/olap/key_coder.cpp:94 6# doris::Field::Field(doris::TabletColumn const&) at /root/src/doris-2.0/be/src/olap/field.h:46 7# doris::FieldFactory::create(doris::TabletColumn const&) in /data/doris/apache-doris-2.0.3-bin-x64/be/lib/doris_be 8# doris::Schema::_init(std::vector<doris::TabletColumn, std::allocatordoris::TabletColumn > const&, std::vector<unsigned int, std::allocator > const&, unsigned long) at /root/src/doris-2.0/be/src/olap/schema.cpp:83 9# doris::Schema::Schema(std::vector<doris::TabletColumn, std::allocatordoris::TabletColumn > const&, std::vector<unsigned int, std::allocator > const&) at /root/src/doris-2.0/be/src/olap/schema.h:108 10# doris::BetaRowsetReader::get_segment_iterators(doris::RowsetReaderContext*, std::vector<std::unique_ptr<doris::RowwiseIterator, std::default_deletedoris::RowwiseIterator >, std::allocator<std::unique_ptr<doris::RowwiseIterator, std::default_deletedoris::RowwiseIterator > > >, bool) in /data/doris/apache-doris-2.0.3-bin-x64/be/lib/doris_be 11# doris::BetaRowsetReader::_init_iterator() at /root/src/doris-2.0/be/src/olap/rowset/beta_rowset_reader.cpp:256 12# doris::BetaRowsetReader::_init_iterator_once() at /root/src/doris-2.0/be/src/olap/rowset/beta_rowset_reader.cpp:251 13# doris::BetaRowsetReader::next_block(doris::vectorized::Block) at /root/src/doris-2.0/be/src/olap/rowset/beta_rowset_reader.cpp:291 14# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() at /root/src/doris-2.0/be/src/vec/olap/vcollect_iterator.cpp:511 15# doris::vectorized::VCollectIterator::Level0Iterator::init(bool) at /root/src/doris-2.0/be/src/vec/olap/vcollect_iterator.cpp:469 16# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptrdoris::RowsetReader, std::allocator<std::shared_ptrdoris::RowsetReader > >&) at /root/src/doris-2.0/be/src/vec/olap/vcollect_iterator.cpp:124 17# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) at /root/src/doris-2.0/be/src/vec/olap/block_reader.cpp:147 18# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /root/src/doris-2.0/be/src/vec/olap/block_reader.cpp:226 19# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /root/src/doris-2.0/be/src/vec/exec/scan/new_olap_scanner.cpp:224 20# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptrdoris::vectorized::VScanner) at /root/src/doris-2.0/be/src/vec/exec/scan/scanner_scheduler.cpp:345 21# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::{lambda()#3}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 22# doris::WorkThreadPool::work_thread(int) at /root/src/doris-2.0/be/src/util/work_thread_pool.hpp:160 23# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84 24# start_thread in /lib64/libpthread.so.0 25# clone in /lib64/libc.so.6

yanjinyin avatar Feb 23 '24 05:02 yanjinyin

Please locate the SQL query statement corresponding to the query_id in fe.audit.log.

vinlee19 avatar Feb 26 '24 06:02 vinlee19

能从日志定位到sql吗?

xy720 avatar Feb 27 '24 11:02 xy720