out of bound in ColumnVector<T>::insert_range_from method (data.size() = 454)

Viewed 93

doris版本:doris-2.0.2-rc05-ae923f7
BE挂了,查询很简单,没有占用大内存,be.out问题日志如下:

INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
F0307 15:29:53.308143  6576 column_vector.cpp:356] Parameters start = 462, length = 8387507244432647080, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 462).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
F0307 15:29:53.308153  6575 column_vector.cpp:356] Parameters start = 887, length = 3689401612825157610, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 887).F0307 15:29:53.308147  6589 column_vector.cpp:356] Parameters start = 442, length = 16690814003432842189, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 442).F0307 15:29:53.308146  6590 column_vector.cpp:356] Parameters start = 364, length = 1, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 364).F0307 15:29:53.352763  6577 column_vector.cpp:356] Parameters start = 896, length = 11367055462188361480, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 896).F0307 15:29:53.352972  6579 column_vector.cpp:356] Parameters start = 414, length = 12675586402999438919, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 414).F0307 15:29:53.356074  6570 column_vector.cpp:356] Parameters start = 398, length = 4995062813679440051, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 398).
*** Check failure stack trace: ***
@     0x557415cf1dd3  doris::vectorized::Channel::add_rows()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf8fe3  doris::vectorized::VDataStreamSender::channel_add_rows<>()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415cf6278  doris::vectorized::VDataStreamSender::send()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d74a5c  doris::pipeline::DataSinkOperator<>::sink()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415d9e88c  doris::pipeline::PipelineTask::execute()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x557415da7103  doris::pipeline::TaskScheduler::_do_work()
@     0x55740f157daf  doris::ThreadPool::dispatch_thread()
@     0x55740f157daf  doris::ThreadPool::dispatch_thread()
@     0x55740f157daf  doris::ThreadPool::dispatch_thread()
@     0x55740f157daf  doris::ThreadPool::dispatch_thread()
@     0x55740f157daf  doris::ThreadPool::dispatch_thread()


@     0x55740f14dd3c  doris::Thread::supervise_thread()
@     0x55740f14dd3c  doris::Thread::supervise_thread()
@     0x55740f14dd3c  doris::Thread::supervise_thread()
@     0x55740f14dd3c  doris::Thread::supervise_thread()
@     0x55740f14dd3c  doris::Thread::supervise_thread()
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc086f92ac3  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@     0x7fc087024850  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
@              (nil)  (unknown)
*** Query id: 15d0206f4637452f-bdb095377b765512 ***
*** tablet id: 0 ***
*** Aborted at 1709796600 (unix time) try "date -d @1709796600" if you are using GNU date ***
*** Current BE git commitID: ae923f7 ***
*** SIGABRT unknown detail explain (@0x18a2) received by PID 6306 (TID 6590 OR 0x7fbf4b8b7640) from PID 6306; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
 1# 0x00007FC086F40520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
 3# raise in /lib/x86_64-linux-gnu/libc.so.6
 4# abort in /lib/x86_64-linux-gnu/libc.so.6
 5# 0x00005574162EFC39 in /opt/apache-doris-2.0.2-bin-x64/be/lib/doris_be
 6# google::LogMessage::SendToLog() in /opt/apache-doris-2.0.2-bin-x64/be/lib/doris_be
 7# google::LogMessage::Flush() in /opt/apache-doris-2.0.2-bin-x64/be/lib/doris_be
 8# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris-2.0.2-bin-x64/be/lib/doris_be
 9# doris::vectorized::ColumnVector<unsigned char>::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /root/src/doris-2.0/be/src/vec/columns/column_vector.cpp:356
10# doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn const&, unsigned long, unsigned long) at /root/src/doris-2.0/be/src/vec/columns/column_nullable.cpp:288
11# doris::vectorized::ColumnMap::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/src/doris-2.0/be/src/vec/columns/column_map.cpp:181
12# doris::vectorized::ColumnMap::insert_indices_from(doris::vectorized::IColumn const&, int const*, int const*) at /root/src/doris-2.0/be/src/vec/columns/column_map.cpp:189
13# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, int const*, int const*) at /root/src/doris-2.0/be/src/vec/core/block.cpp:938
14# doris::vectorized::Channel::add_rows(doris::vectorized::Block*, std::vector<int, std::allocator<int> > const&) at /root/src/doris-2.0/be/src/vec/sink/vdata_stream_sender.cpp:227
15# doris::Status doris::vectorized::VDataStreamSender::channel_add_rows<std::vector<doris::vectorized::Channel*, std::allocator<doris::vectorized::Channel*> > >(doris::RuntimeState*, std::vector<doris::vectorized::Channel*, std::allocator<doris::vectorized::Channel*> >&, int, unsigned long const*, int, doris::vectorized::Block*) at /root/src/doris-2.0/be/src/vec/sink/vdata_stream_sender.h:404
16# doris::vectorized::VDataStreamSender::send(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/src/doris-2.0/be/src/vec/sink/vdata_stream_sender.cpp:672
17# doris::pipeline::DataSinkOperator<doris::pipeline::ExchangeSinkOperatorBuilder>::sink(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState) at /root/src/doris-2.0/be/src/pipeline/exec/operator.h:281
18# doris::pipeline::PipelineTask::execute(bool*) at /root/src/doris-2.0/be/src/pipeline/pipeline_task.cpp:262
19# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/src/doris-2.0/be/src/pipeline/task_scheduler.cpp:274
20# doris::ThreadPool::dispatch_thread() in /opt/apache-doris-2.0.2-bin-x64/be/lib/doris_be
21# doris::Thread::supervise_thread(void*) at /root/src/doris-2.0/be/src/util/thread.cpp:470
22# 0x00007FC086F92AC3 in /lib/x86_64-linux-gnu/libc.so.6
23# 0x00007FC087024850 in /lib/x86_64-linux-gnu/libc.so.6
start time: Thu Mar  7 15:36:48 CST 2024
INFO: java_cmd /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
F0307 16:14:09.723127 66401 column_vector.cpp:356] Parameters start = 453, length = 3862361327, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 453).
*** Check failure stack trace: ***
    @     0x564737eecc59  google::LogMessageFatal::~LogMessageFatal()
    @     0x564732e5fc08  doris::vectorized::ColumnVector<>::insert_range_from()
    @     0x564732e31567  doris::vectorized::ColumnNullable::insert_range_from()
F0307 16:14:09.751727 66406 column_vector.cpp:356] Parameters start = 454, length = 8529663114216978721, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 454).
*** Check failure stack trace: ***
    @     0x564732e2c31d  doris::vectorized::ColumnMap::insert_from()
F0307 16:14:09.751727 66406 column_vector.cpp:356] Parameters start = 454, length = 8529663114216978721, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 454).F0307 16:14:09.753540 66412 column_vector.cpp:356] Parameters start = 427, length = 7310291539315811265, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 427).
*** Check failure stack trace: ***

1 Answers
  1. grep "15d0206f4637452f-bdb095377b765512" fe/log/fe.audit.log 看下是哪个查询导致的?

  2. 看着是和pipeline 有关系,把pipeline 关了再查询一下试试的,可以session 级别关闭测试下:
    set experimental_enable_pipeline_engine = false; 同一个session下跑一下sql 试试