[INTERNAL_ERROR][INTERNAL_ERROR]close wait failed coz rpc error. [INTERNAL_ERROR]tablet error: [DATA_QUALITY_ERROR]Reached max column size limit 4096

Viewed 44

各位社区大佬们好,在基于Doris 做日志分析平台时遇到以下问题;通过stream load 写入数据报错
doris streamload 报错
[INTERNAL_ERROR][INTERNAL_ERROR]close wait failed coz rpc error. [INTERNAL_ERROR]tablet error: [DATA_QUALITY_ERROR]Reached max column size limit 4096

doris 版本 3.0
建表语句
CREATE TABLE dev_event_data
(
event_time DATETIME COMMENT '事件时间',
event_name VARCHAR(255) COMMENT '事件名称',
event_type VARCHAR(255) COMMENT '事件类型',
event_host VARCHAR(20) COMMENT '事件主机',
user_id VARCHAR(64) COMMENT '用户ID',
properties variant COMMENT '事件属性',
collection variant COMMENT '采集器源数据',
collection_time DATETIME(3) COMMENT '数据写入时间',
INDEX idx_event_name (event_name) USING INVERTED,
INDEX idx_event_time (event_time) USING INVERTED,
INDEX idx_event_host (event_host) USING INVERTED,
INDEX idx_collection_time (collection_time) USING INVERTED,
INDEX idx_collection (collection) USING INVERTED,
INDEX idx_properties (properties) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
)
ENGINE = OLAP
DUPLICATE KEY(event_time)
PARTITION BY RANGE(event_time) ()
DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
"compression" = "zstd",
"compaction_policy" = "time_series",
"dynamic_partition.enable" = "true",
"dynamic_partition.create_history_partition" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.start" = "-365",
"dynamic_partition.end" = "400",
"dynamic_partition.prefix" = "p",
"dynamic_partition.buckets" = "60",
"dynamic_partition.replication_num" = "2",
"replication_num" = "2",
"storage_policy" = "dev_event_15day"
);

展开字段后查看表字段数量才300个
set describe_extend_variant_column = true;
DESC dev_event_data;

以上问题该如何解决呢,烦请指教

1 Answers

可以适当调整 BE 配置 variant_max_merged_tablet_schema_size的大小,这里如果实际没有这么多列,可能是切分问题,你这里先修改上面的参数