对Doris数仓进行数据迁移,使用stream load方式写入数据,在观察写入日志时发现迁移的速度会出现单笔数据写入长时间的任务,导致数据迁移速度效率很慢,以下是几个doris返回的数据信息:
Doris返回信息:{ "TxnId": 1219317, "Label": "audit_20240517_173707_3ece0e0e43994a42a25663a56c0a464b", "Comment": "", "TwoPhaseCommit": "false", "Status": "Success", "Message": "OK", "NumberTotalRows": 39726, "NumberLoadedRows": 39726, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10798217, "LoadTimeMs": 4857, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 4, "ReadDataTimeMs": 4, "WriteDataTimeMs": 4791, "CommitAndPublishTimeMs": 60 }
组装请求和发送写入请求到返回的耗时:9202毫秒 ;
还会出现几十秒或者几十分钟的这些日志信息,请教下出现这些情况的原因会有那些?
补充:
出现日志信息,Doris返回信息:{ "TxnId": 1276513, "Label": "audit_20240520_163137_6e5e0bae157d4ba2bf5df6379947925b", "Comment": "", "TwoPhaseCommit": "false", "Status": "Success", "Message": "OK", "NumberTotalRows": 10000, "NumberLoadedRows": 10000, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 9317985, "LoadTimeMs": 629, "BeginTxnTimeMs": 1, "StreamLoadPutTimeMs": 5, "ReadDataTimeMs": 7, "WriteDataTimeMs": 571, "CommitAndPublishTimeMs": 51 }
整个过程耗时:930681毫秒