Doris 2.1.3版本,3个FE, 3个BE,FE可以连续稳定运行,但三个BE节点每运行24小时就会出现宕机,开始认为是系统连接限制,后来改为system service,但还是24小时出现宕机的问题。下面是2次出现问题后的be.out文件的输出:
start time: Tue Jun 25 23:04:38 CST 2024
INFO: java_cmd /data1/jdk1.8.0_202//bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 10356 ***
*** Aborted at 1719414448 (unix time) try "date -d @1719414448" if you are using GNU date ***
*** Current BE git commitID: 2dc65ce356 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 17371 (TID 18521 OR 0x7f114dfd9700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo*, void*) in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
4# 0x00007F13CD4F4280 in /lib64/libc.so.6
5# doris::segment_v2::SegmentWriter::_full_encode_keys[abi:cxx11](std::vector<doris::KeyCoder const*, std::allocator<doris::KeyCoder const*> > const&, std::vector<doris::vectorized::IOlapColumnDataAccessor*, std::allocator > const&, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:913
6# doris::segment_v2::SegmentWriter::_generate_short_key_index(std::vector<doris::vectorized::IOlapColumnDataAccessor*, std::allocator >&, unsigned long, std::vector<unsigned long, std::allocator > const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1297
7# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block const*, unsigned long, unsigned long) in /data1/doris/be/lib/doris_be
8# doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector<unsigned int, std::allocator > const&, bool, unsigned int) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:85
9# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector<unsigned int, std::allocator > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator >) in /data1/doris/be/lib/doris_be
10# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:383
11# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:371
12# doris::Compaction::do_compaction(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:136
13# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/olap/base_compaction.cpp:79
14# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:118
15# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:1947
16# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr, doris::CompactionType, bool)::$_1>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
17# doris::ThreadPool::dispatch_thread() in /data1/doris/be/lib/doris_be
18# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
19# start_thread in /lib64/libpthread.so.0
20# __clone in /lib64/libc.so.6
start time: Fri Jun 28 22:05:59 CST 2024
INFO: java_cmd /data1/jdk1.8.0_202//bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 10356 ***
*** Aborted at 1719670546 (unix time) try "date -d @1719670546" if you are using GNU date ***
*** Current BE git commitID: 2dc65ce356 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 10017 (TID 11345 OR 0x7f112f771700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo*, void*) in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /data1/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
4# 0x00007F13A78CF280 in /lib64/libc.so.6
5# doris::segment_v2::SegmentWriter::_full_encode_keys[abi:cxx11](std::vector<doris::KeyCoder const*, std::allocator<doris::KeyCoder const*> > const&, std::vector<doris::vectorized::IOlapColumnDataAccessor*, std::allocator > const&, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:913
6# doris::segment_v2::SegmentWriter::_generate_short_key_index(std::vector<doris::vectorized::IOlapColumnDataAccessor*, std::allocator >&, unsigned long, std::vector<unsigned long, std::allocator > const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1297
7# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block const*, unsigned long, unsigned long) in /data1/doris/be/lib/doris_be
8# doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const*, std::vector<unsigned int, std::allocator > const&, bool, unsigned int) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:85
9# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector<unsigned int, std::allocator > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator >) in /data1/doris/be/lib/doris_be
10# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:383
11# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:371
12# doris::Compaction::do_compaction(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:136
13# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/olap/base_compaction.cpp:79
14# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:118
15# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:1947
16# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr, doris::CompactionType, bool)::$_1>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
17# doris::ThreadPool::dispatch_thread() in /data1/doris/be/lib/doris_be
18# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
19# start_thread in /lib64/libpthread.so.0
20# __clone in /lib64/libc.so.6
请帮助指教,谢谢!