Doris通过spark-doris-conn将hive表导入doris，导致BE节点Crash

Question

1 环境

doris 2.0.3、2.1.5两个版本都有问题
spark 3.1.1
spark-doris-conn 最新

2 spark-doris-conn将hive表中计算好的bitmap结果导入doris，导致BE节点Crash

2.1 明确定位到问题，百分百复现

hive建表采用官网说的用binary存储，textfile存储（官网链接）
spark-doris-conn导入到doris用下面就会有问题
- 如果采用"doris.write.fields"="xxx,user_id,user_id=bitmap_from_base64(user_id)"这种方式导入就会报错：
- 如果采用"doris.write.fields"="xxx,user_id,user_id=bitmap_from_base64(to_base64(user_id))"这种方式，就会导入一会就会直接将全部BE crash，可以参考下面的 3 be.out

2.2 可以换种方式避免

hive建表用String类型存储，spark计算时用base64(to_bitmap(field)) 函数存到表里，hive建表用parquet存储，然后导入doris参数用"doris.write.fields"="xxx,user_id,user_id=bitmap_from_base64(user_id)"即可

3 be.out


start time: Fri Jul 26 03:04:13 UTC 2024
INFO: java_cmd /usr/local/openjdk-8//bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0726 03:04:17.395560   136 block_file_cache.cpp:219] max file reader cache size is: 349525, resource hard limit is: 1048576, config file_cache_max_file_reader_cache_size is: 1000000
I0726 03:04:17.395808   164 block_lru_file_cache.cpp:111] file cache path=/opt/apache-doris/be/filecache, disposable queue size=10737418240 elements=102400, index queue size=5368709120 elements=102400, query queue size=91268055040 elements=102400
I0726 03:04:19.664393   164 block_lru_file_cache.cpp:141] After initialize file cache path=/opt/apache-doris/be/filecache, disposable queue size=0 elements=0, index queue size=0 elements=0, query queue size=67819177918 elements=19890, init cost(ms)=2268
I0726 03:04:19.664426   164 block_file_cache_factory.cpp:87] [FileCache] path: /opt/apache-doris/be/filecache total_size: 107374182400
terminate called after throwing an instance of 'std::runtime_error'
  what():  failed alloc while reading
*** Query id: 1b4a8f5303fba682-64257504924fb683 ***
*** tablet id: 0 ***
*** Aborted at 1721963551 (unix time) try "date -d @1721963551" if you are using GNU date ***
*** Current BE git commitID: 2a5949cb9e ***
*** SIGABRT unknown detail explain (@0x88) received by PID 136 (TID 637 OR 0x7fbacab54700) from PID 136; stack trace: ***
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:417
 1# 0x00007FBBF5552D60 in /lib/x86_64-linux-gnu/libc.so.6
 2# raise in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 6# 0x000055C65C96E751 in /opt/apache-doris/be/lib/doris_be
 7# 0x000055C65C96E8A4 in /opt/apache-doris/be/lib/doris_be
 8# roaring::Roaring::read(char const*, bool) in /opt/apache-doris/be/lib/doris_be
 9# doris::detail::Roaring64Map::read(char const*) in /opt/apache-doris/be/lib/doris_be
10# doris::BitmapValue::deserialize(char const*) at /home/zcp/repo_center/doris_release/doris/be/src/util/bitmap_value.h:2264
11# doris::vectorized::BitmapFromBase64::vector(doris::vectorized::PODArray, 15ul, 16ul> const&, doris::vectorized::PODArray, 15ul, 16ul> const&, std::vector >&, doris::vectorized::PODArray, 15ul, 16ul>&, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function_bitmap.cpp:298
12# doris::vectorized::FunctionBitmapAlwaysNull::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function_bitmap.cpp:373
13# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.h:521
14# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.cpp:153
15# doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.cpp:245
16# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.cpp:268
17# doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.cpp:274
18# doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/functions/function.h:183
19# doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exprs/vectorized_fn_call.cpp:158
20# doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exprs/vexpr_context.cpp:60
21# doris::vectorized::VFileScanner::_convert_to_output_block(doris::vectorized::Block*) in /opt/apache-doris/be/lib/doris_be
22# doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vfile_scanner.cpp:312
23# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vfile_scanner.cpp:245
24# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /opt/apache-doris/be/lib/doris_be
25# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:387
26# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
27# doris::WorkThreadPool::work_thread(int) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:160
28# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
29# start_thread in /lib/x86_64-linux-gnu/libpthread.so.0
30# __clone in /lib/x86_64-linux-gnu/libc.so.6

徐振超@SelectDB · Accepted Answer

已知问题，修复pr https://github.com/apache/doris/pull/30773
推荐升级到2.0.x的最新版本