【已解决】客户端查询引起 flink 同步 doris 异常

Viewed 72

一个与 flink 同步无关的查询(navicat 中执行的)引起了 flink 同步错误, 查询语句如下:

 (select 	COUNT(*) cnt    ,INPUTDATE   from jdbc_bi.prod.prpphead t1 
 where t1.inputdate  BETWEEN TIMESTAMP'2024-04-23 00:00:00'   and CURDATE() and t1.UNDERWRITEFLAG in ('1','3') group by INPUTDATE) a
 left join 
 ( select 	COUNT(*) cnt2  , INPUTDATE as  inputdate2 from prod.prpphead t2
 where t2.inputdate  BETWEEN TIMESTAMP'2024-04-23 00:00:00' and CURDATE() and t2.UNDERWRITEFLAG in ('1','3')  group by inputdate) b on  a.inputdate = b.inputdate2) order by inputdate;

其中 jdbc_bi.prod.prpphead 为外部表;
最终重启 doris 后问题没有重现,怀疑有死锁类的问题

--------------异常日志如下
2024-04-28 08:50:57,488 INFO  org.apache.doris.flink.sink.writer.DorisStreamLoad           [] - load Result {
    "TxnId": 5401730,
    "Label": "coredate-b3434c08-3a9b-4206-bef8-978b92f9c527_prod_prpphead_0_1_dc9e751f-3e60-40a5-a421-3bbb53fd96a3",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Unknown column 'INPUTDATE' in 't2'\n\n\t0#  doris::Status doris::Status::create<true>(doris::TStatus const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187\n\t1#  doris::StreamLoadAction::_process_put(doris::HttpRequest*, std::shared_ptr<doris::StreamLoadContext>) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:449\n\t2#  doris::StreamLoadAction::_on_header(doris::HttpRequest*, std::shared_ptr<doris::StreamLoadContext>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701\n\t3#  doris::StreamLoadAction::on_header(doris::HttpRequest*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:345\n\t4#  doris::EvHttpServer::on_header(evhttp_request*) at /home/zcp/repo_center/doris_release/doris/be/src/http/ev_http_server.cpp:255\n\t5#  ?\n\t6#  bufferevent_run_readcb_\n\t7#  ?\n\t8#  ?\n\t9#  ?\n\t10# ?\n\t11# std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0\n\t13# doris::Thread::supervise_thread(void*) at /var/local/ldb_toolchain/bin/../usr/include/pthread.h:562\n\t14# start_thread\n\t15# __clone\n",
    "NumberTotalRows": 0,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 0,
    "LoadTimeMs": 0,
    "BeginTxnTimeMs": 2,
    "StreamLoadPutTimeMs": 2,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 0,
    "CommitAndPublishTimeMs": 0
}
1 Answers

目前是什么版本,能否再现?

基于当前异常信息报的 Unknown column 'INPUTDATE' ,可能存在schema变更或元数据不同步的偶发情况。