routine load在解析kafka数据时,需要依赖解析出来的字段衍生出新的字段,但是最终又不需要这个解析字段,怎么处理
routine load在解析kafka数据时,需要依赖解析出来的字段衍生出新的字段,但是最终又不需要这个解析字段,怎么处理
以官网建表例子举例,假设k1,k2,k3,v2,v3都是doris表实际存在的列,v1是需要过滤的列,且doris表中没有这个列名,可以通过这种方式过滤掉v1,因为不存在这个列,v3又可以通过v1这个列来计算
CREATE ROUTINE LOAD example_db.test1 ON example_tbl
COLUMNS TERMINATED BY ",",
COLUMNS(k1, k2, k3, v1, v2, v3 = v1 * 100)
PROPERTIES
(
"desired_concurrent_number"="3",
"max_batch_interval" = "20",
"max_batch_rows" = "300000",
"max_batch_size" = "209715200",
"strict_mode" = "false"
)
FROM KAFKA
(
"kafka_broker_list" = "broker1:9092,broker2:9092,broker3:9092",
"kafka_topic" = "my_topic",
"property.group.id" = "xxx",
"property.client.id" = "xxx",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);