使用Streamload将s3的数据导入到Doris,导入了几分钟后开始报错

Viewed 71

一开始导入很正常,但是几分钟后突然报错,重试也是一样,都是到一定时间就报错,从日志拉出来的报错信息如下所示

W20241011 20:39:55.375949 3434878 file_reader.cpp:36] [INTERNAL_ERROR]cancelled: sender is gone
W20241011 20:39:55.376004 3434878 scanner_scheduler.cpp:272] Scan thread read VScanner failed: [INTERNAL_ERROR]cur path: . cancelled: sender is gone
W20241011 20:39:55.376077 3430725 task_scheduler.cpp:361] Pipeline task failed. query_id: 32417db0edce035f-a03854ee94713aae|32417db0edce035f-a03854ee94713aaf reason: [INTERNAL_ERROR]cur path: . cancelled: sender is gone
W20241011 20:39:55.376143 3430725 pipeline_fragment_context.cpp:191] PipelineFragmentContext 32417db0edce035f-a03854ee94713aae|32417db0edce035f-a03854ee94713aaf is canceled, cancel message: cur path: . cancelled: sender is gone
W20241011 20:39:55.376176 3425565 vtablet_writer.cpp:587] cancel node channel VNodeChannel[81084-10006], load_id=32417db0edce035f-a03854ee94713aae, txn_id=186356, node=10.100.5.234:8060, error message: [CANCELLED]cur path: . cancelled: sender is gone
W20241011 20:39:55.376253 3425565 vtablet_writer.cpp:587] cancel node channel VNodeChannel[81084-10005], load_id=32417db0edce035f-a03854ee94713aae, txn_id=186356, node=10.100.5.233:8060, error message: [CANCELLED]cur path: . cancelled: sender is gone
W20241011 20:39:55.376278 3425565 vtablet_writer.cpp:587] cancel node channel VNodeChannel[81084-10004], load_id=32417db0edce035f-a03854ee94713aae, txn_id=186356, node=10.100.5.235:8060, error message: [CANCELLED]cur path: . cancelled: sender is gone
W20241011 20:39:55.376330 3425611 fragment_mgr.cpp:538] report error status: cur path: . cancelled: sender is gone to coordinator: TNetworkAddress(hostname=10.100.5.231, port=9020), query id: 32417db0edce035f-a03854ee94713aae, instance id: 32417db0edce035f-a03854ee94713aaf
W20241011 20:39:55.379689 3425565 stream_load_executor.cpp:99] fragment execute failed, err_msg=[CANCELLED]cur path: . cancelled: sender is gone, id=32417db0edce035f-a03854ee94713aae, job_id=-1, txn_id=186356, label=seatunnel-doris-1f76ae7f-4bfc-49ff-a835-abf3fb4235a8_897100148105019425_0_1728650144730, elapse(s)=250
W20241011 20:39:55.728089 3426132 status.h:413] meet error status: [INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::LoadChannelMgr::_get_load_channel(std::shared_ptr&, bool&, doris::UniqueId const&, doris::PTabletWriterAddBlockRequest const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# std::_Function_handler<void (), doris::PInternalServiceImpl::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
3# doris::WorkThreadPool::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
4# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
5# ?
6# clone
W20241011 20:39:55.728154 3426132 internal_service.cpp:470] tablet writer add block failed, message=[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::LoadChannelMgr::_get_load_channel(std::shared_ptr&, bool&, doris::UniqueId const&, doris::PTabletWriterAddBlockRequest const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# std::_Function_handler<void (), doris::PInternalServiceImpl::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
3# doris::WorkThreadPool::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
4# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
5# ?
6# clone
, id=32417db0edce035f-a03854ee94713aae, index_id=81084, sender_id=0, backend id=10004
W20241011 20:39:55.742434 3426525 status.h:432] meet error status: [INTERNAL_ERROR]PStatus: (10.100.5.235)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::Status doris::Status::create(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# void doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::_process_status(doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::Run() at /home/zcp/repo_center/doris_release/doris/be/src/util/ref_count_closure.h:91
3# brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
4# brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*)
5# brpc::ProcessInputMessage(void*)
6# brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
7# brpc::InputMessenger::OnNewMessages(brpc::Socket*)
8# brpc::Socket::ProcessEvent(void*)
9# bthread::TaskGroup::task_runner(long)
10# bthread_make_fcontext
W20241011 20:39:55.742441 3426540 status.h:432] meet error status: [INTERNAL_ERROR]PStatus: (10.100.5.234)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::Status doris::Status::create(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# void doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::_process_status(doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::Run() at /home/zcp/repo_center/doris_release/doris/be/src/util/ref_count_closure.h:91
3# brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
4# brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*)
5# brpc::ProcessInputMessage(void*)
6# brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
7# brpc::InputMessenger::OnNewMessages(brpc::Socket*)
8# brpc::Socket::ProcessEvent(void*)
9# bthread::TaskGroup::task_runner(long)
10# bthread_make_fcontext
W20241011 20:39:55.742475 3426525 ref_count_closure.h:119] RPC meet error status: [INTERNAL_ERROR]PStatus: (10.100.5.235)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::Status doris::Status::create(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# void doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::_process_status(doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::Run() at /home/zcp/repo_center/doris_release/doris/be/src/util/ref_count_closure.h:91
3# brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
4# brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*)
5# brpc::ProcessInputMessage(void*)
6# brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
7# brpc::InputMessenger::OnNewMessages(brpc::Socket*)
8# brpc::Socket::ProcessEvent(void*)
9# bthread::TaskGroup::task_runner(long)
10# bthread_make_fcontext
W20241011 20:39:55.742507 3426540 ref_count_closure.h:119] RPC meet error status: [INTERNAL_ERROR]PStatus: (10.100.5.234)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=32417db0edce035f-a03854ee94713aae
0# doris::Status doris::Status::create(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# void doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::_process_status(doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
2# doris::AutoReleaseClosure<doris::PTabletWriterAddBlockRequest, doris::vectorized::WriteBlockCallback >::Run() at /home/zcp/repo_center/doris_release/doris/be/src/util/ref_count_closure.h:91
3# brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
4# brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*)
5# brpc::ProcessInputMessage(void*)
6# brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
7# brpc::InputMessenger::OnNewMessages(brpc::Socket*)
8# brpc::Socket::ProcessEvent(void*)
9# bthread::TaskGroup::task_runner(long)
10# bthread_make_fcontext

1 Answers

方便提供一下复现步骤吗,表结构和数据以及操作。