Doris 单个shell窗口执行insert into select语句,在频繁执行几次后insert语句会hang住,复现一般等待30s左右会接触hang,目前可知是获取table元数据锁过程产生等待

Viewed 28

Drosi版本:2.1.1-rc05;
建表语句:目标表记录数不超过1000行数据;原表1000行数据;
CREATE TABLE tbl_target (
key1 VARCHAR(30) NOT NULL COMMENT 'key1',
key2 VARCHAR(30) NOT NULL COMMENT 'key2',
value1 VARCHAR(30) NULL DEFAULT "" COMMENT '',
value2 INT NULL COMMENT '',
insertTime DATETIME NULL DEFAULT CURRENT_TIMESTAMP COMMENT '',
updateTime DATETIME NULL DEFAULT CURRENT_TIMESTAMP COMMENT '',
state1 BOOLEAN NULL DEFAULT "0" COMMENT '',
state2 INT NULL COMMENT '',
state3 INT NULL DEFAULT "1" COMMENT ''
) ENGINE=OLAP
UNIQUE KEY(key1, key2)
COMMENT ''
DISTRIBUTED BY HASH(key1) BUCKETS 3
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V1",
"enable_unique_key_merge_on_write" = "true",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);

测试执行语句:
insert into tbl_target(
key1,
key2,
updateTime,
state1
)

select
batchNo,
taxNo,
now() as updateTime,
'8' as state1
from dwd.dwd_table1
where key2 in (
select key2
from dwd.dwd_table2
order by key2 desc
limit 5
);

执行方式:doris shell复制粘贴

现象:当快速复制粘贴10次或更多insert into语句到shell,可能在第8次insert 卡住,本批次插入时间一般在0.1秒内返回,但卡住这条SQL通常会在30s返回结果,初步猜测是存在锁竞争,在调整过参数后等待可以通过,但比较慢;调参前会触发报错:get tableList write timeout 错误。

数据量并不大,单开shell窗口个人理解不存在并发冲突问题?
compaction导致插入阻塞?元数据更新代码存在性能问题?
doris客户端insert into执行方式底层原理?
doris版本存在稳定性问题?

了解大佬给点排查意见

1 Answers

doris并不建议频繁的insert into导入,如果方便的话尽量攒批导入,或者通过group commit来高并发小写入。