【已解决】数据迁移性能问题

Viewed 97

对Doris数仓进行数据迁移,使用stream load方式写入数据,在观察写入日志时发现迁移的速度会出现单笔数据写入长时间的任务,导致数据迁移速度效率很慢,以下是几个doris返回的数据信息:

Doris返回信息:{ "TxnId": 1219317, "Label": "audit_20240517_173707_3ece0e0e43994a42a25663a56c0a464b", "Comment": "", "TwoPhaseCommit": "false", "Status": "Success", "Message": "OK", "NumberTotalRows": 39726, "NumberLoadedRows": 39726, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10798217, "LoadTimeMs": 4857, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 4, "ReadDataTimeMs": 4, "WriteDataTimeMs": 4791, "CommitAndPublishTimeMs": 60 }

组装请求和发送写入请求到返回的耗时:9202毫秒 ;
还会出现几十秒或者几十分钟的这些日志信息,请教下出现这些情况的原因会有那些?


补充:
出现日志信息,Doris返回信息:{ "TxnId": 1276513, "Label": "audit_20240520_163137_6e5e0bae157d4ba2bf5df6379947925b", "Comment": "", "TwoPhaseCommit": "false", "Status": "Success", "Message": "OK", "NumberTotalRows": 10000, "NumberLoadedRows": 10000, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 9317985, "LoadTimeMs": 629, "BeginTxnTimeMs": 1, "StreamLoadPutTimeMs": 5, "ReadDataTimeMs": 7, "WriteDataTimeMs": 571, "CommitAndPublishTimeMs": 51 }
整个过程耗时:930681毫秒

1 Answers

【问题状态】已处理
【问题处理】修改 be.conf 中参数 sync_tablet_meta = false,结合集群情况调整并发数