mysql数据导入doris执行失败

Viewed 26

doris版本:2.1.7

doris资源大小:fe 4c/8G,be:8C/32G
执行sql

set batch_size = 500;
insert into internal.dwd.`prod_jira_issue`
select
    pk_id,
    ...
from mysql8.dqa.prod_jira_issue;

数据量只有69万条,大小2G,但是报错(若加上limit 则能成功),查看sql审计日志,找到错误信息:

errCode = 2, detailMessage = (doriscluster-be-1.doriscluster-be-internal.doris.svc.cluster.local)[CANCELLED]Encountered unqualified data, stop processing. url: http://doriscluster-be-1.doriscluster-be-internal.doris.svc.cluster.local:30840/api/_load_error_log?file=__shard_2/error_log_insert_stmt_1701e6403c0d4d03-a8c826471c6230ee_1701e6403c0d4d03_a8c826471c6230ee

再查看be-1的be.WARING日志,中报错:

data, stop processing
RuntimeLogger W20250410 15:38:57.105715   637 load_stream_stub.cpp:371] LoadStreamStub load_id=1701e6403c0d4d03-a8c826471c6230ed, src_id=10007, dst_id=10007, stream_id=25769824369 is cancelled because of [DATA_QUALITY_ERROR]Encountered unqualified data, stop processing
RuntimeLogger W20250410 15:39:01.353911  1744 task_scheduler.cpp:361] Pipeline task failed. query_id: 1701e6403c0d4d03-a8c826471c6230ed|0-0 reason: [DATA_QUALITY_ERROR]Encountered unqualified data, stop processing
RuntimeLogger W20250410 15:39:01.354102  1744 pipeline_x_fragment_context.cpp:168] PipelineXFragmentContext cancel instance: 1701e6403c0d4d03-a8c826471c6230ee
RuntimeLogger W20250410 15:39:01.354285   678 fragment_mgr.cpp:549] report error status: Encountered unqualified data, stop processing to coordinator: TNetworkAddress(hostname=doriscluster-fe-2.doriscluster-fe-internal.doris.svc.cluster.local, port=9020), query id: 1701e6403c0d4d03-a8c826471c6230ed, instance id: 0-0
RuntimeLogger W20250410 15:39:02.042217 312915 scanner_scheduler.cpp:292] Scan thread read VScanner failed: [CANCELLED]cancelled
RuntimeLogger W20250410 15:40:15.519420   704 status.h:413] meet error status: [TIMEOUT]Query tiemout

        0#  doris::ResultBufferMgr::cancel_thread() at /home/zcp/repo_center/doris_release/doris/be/src/runtime/result_buffer_mgr.cpp:210
        1#  doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        2#  ?
        3#  ?
RuntimeLogger W20250410 15:40:53.522161   704 status.h:413] meet error status: [TIMEOUT]Query tiemout

        0#  doris::ResultBufferMgr::cancel_thread() at /home/zcp/repo_center/doris_release/doris/be/src/runtime/result_buffer_mgr.cpp:210
        1#  doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562

be与be状态都正常:
image.png

image.png

请问一下可能会是什么原因,是需要设置什么参数么?

image.png

1 Answers

已解决,因为,直接看返回的这个路径中的错误就好了:http://doriscluster-be-1.doriscluster-be-internal.doris.svc.cluster.local:30840/api/_load_error_log?file=__shard_2/error_log_insert_stmt_1701e6403c0d4d03-a8c826471c6230ee_1701e6403c0d4d03_a8c826471c6230ee

因为我是云环境,内部地址无法打开,所以直接去看的日志文件,日志文件报超时,被误导了

直接进pod,curl上面这个路径就行了