【已解决】doris 使用routine load解析kafka数据,导入doris时需要过滤掉一些字段,但是有些衍生字段又是依赖这个需要过滤掉的字段,该如何处理

Viewed 31

routine load在解析kafka数据时,需要依赖解析出来的字段衍生出新的字段,但是最终又不需要这个解析字段,怎么处理

1 Answers

以官网建表例子举例,假设k1,k2,k3,v2,v3都是doris表实际存在的列,v1是需要过滤的列,且doris表中没有这个列名,可以通过这种方式过滤掉v1,因为不存在这个列,v3又可以通过v1这个列来计算

CREATE ROUTINE LOAD example_db.test1 ON example_tbl
        COLUMNS TERMINATED BY ",",
        COLUMNS(k1, k2, k3, v1, v2, v3 = v1 * 100)
        PROPERTIES
        (
            "desired_concurrent_number"="3",
            "max_batch_interval" = "20",
            "max_batch_rows" = "300000",
            "max_batch_size" = "209715200",
            "strict_mode" = "false"
        )
        FROM KAFKA
        (
            "kafka_broker_list" = "broker1:9092,broker2:9092,broker3:9092",
            "kafka_topic" = "my_topic",
            "property.group.id" = "xxx",
            "property.client.id" = "xxx",
            "property.kafka_default_offsets" = "OFFSET_BEGINNING"
        );