paimon2doris insert into select xxx OutOfMemoryError

Viewed 76

doris2.0.14,paimon0.8 通过外部连接 INSERT INTO center_ods.ods_ra_customer_trx_lines_all
select * from paimon_catalog.xx.xx

抛出cur path: hdfs://dummyPath. failed to init reader for file hdfs://dummyPath, err: [INTERNAL_ERROR]OutOfMemoryError: GC overhead limit exceeded
表的数据量2600w+行

[1105] [HY000]: errCode = 2, detailMessage = (10.91.11.25)[INTERNAL_ERROR]cur path: hdfs://dummyPath. failed to init reader for file hdfs://dummyPath, err: [INTERNAL_ERROR]OutOfMemoryError: GC overhead limit exceeded

0#  doris::JniUtil::GetJniExceptionMsg(JNIEnv_*, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/util/jni-util.h:117
1#  doris::vectorized::JniConnector::open(doris::RuntimeState*, doris::RuntimeProfile*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
2#  doris::vectorized::PaimonJniReader::init_reader(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, 
1 Answers

这个是BE的jvm oom 了。
修改下 be.conf 里面的 jvm 参数为 -Xmx4096m
image.png