2.0.5版本升级到2.1.1版本后，原来的有物化视图的表无法通过spark导入数据

Viewed 67
2.1 upgrade

报错信息：

Caused by: org.apache.doris.spark.exception.StreamLoadException: stream load error: status: 200, resp msg: OK, resp content: {
    "TxnId": 13258230,
    "Label": "spark_streamload_20240409_101354_9b6132b70ce249a7a2455366fecad459",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = column has no source field, column=mva_GENERIC_AGGREGATION__max_by_state(`tag_value`, `create_ts`)\n\n\t0#  doris::Status doris::Status::create<true>(doris::TStatus const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187\n\t1#  doris::StreamLoadAction::_process_put(doris::HttpRequest*, std::shared_ptr<doris::StreamLoadContext>) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:449\n\t2#  doris::StreamLoadAction::_on_header(doris::HttpRequest*, std::shared_ptr<doris::StreamLoadContext>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701\n\t3#  doris::StreamLoadAction::on_header(doris::HttpRequest*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:345\n\t4#  doris::EvHttpServer::on_header(evhttp_request*) at /home/zcp/repo_center/doris_release/doris/be/src/http/ev_http_server.cpp:255\n\t5#  ?\n\t6#  bufferevent_run_readcb_\n\t7#  ?\n\t8#  ?\n\t9#  ?\n\t10# ?\n\t11# std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0\n\t13# doris::Thread::supervise_thread(void*) at /var/local/ldb_toolchain/bin/../usr/include/pthread.h:562\n\t14# start_thread\n\t15# clone\n",

我新建了物化视图，对比后发现聚合类型的字段名称有差异
2.0.5版本的物化视图字段：
image.png
2.1.1版本的物化视图字段：
image.png

zhangjj77
2 Answers

哈喽，麻烦贴一下，您的表结构和创建物化视图的语句，我们来复现下看看的

阿渊@SelectDB5530

建表SQL

CREATE TABLE `test` (
  `pk_id` VARCHAR(64) NOT NULL,
  `collect_date` DATE NOT NULL COMMENT '采集日期',
  `collect_timestamp` DATETIME NULL COMMENT '采集时间戳',
  `create_ts` DATETIME NULL,
  `sys_type` VARCHAR(8) NULL COMMENT '系统类型',
  `source_type` VARCHAR(8) NULL COMMENT '来源系统',
  `message_type` VARCHAR(8) NULL COMMENT '消息类型',
  `id` BIGINT NOT NULL COMMENT '项目主键',
  `code` VARCHAR(128) NOT NULL COMMENT '项目编码',
  `room_id` BIGINT NOT NULL COMMENT '站点主键',
  `room_code` VARCHAR(128) NOT NULL COMMENT '站点编码',
  `room_name` VARCHAR(256) NULL COMMENT '站点名称',
  `item_id` BIGINT NULL COMMENT '回路主键',
  `item_code` VARCHAR(128) NOT NULL COMMENT '回路编码',
  `item_name` VARCHAR(256) NULL COMMENT '回路名称',
  `target_code` VARCHAR(68) NOT NULL COMMENT '指标编码',
  `target_type_code` VARCHAR(68) NULL COMMENT '指标类型编码',
  `target_group` VARCHAR(68) NULL COMMENT '指标分组编码',
  `first_name` VARCHAR(68) NULL,
  `second_name` VARCHAR(68) NULL,
  `bm_name` VARCHAR(68) NULL,
  `reason` VARCHAR(8) NULL,
  `tag_value` DECIMAL(20, 4) NULL COMMENT '值',
  `collect_ts` VARCHAR(32) NULL COMMENT '采集时间戳',
  `collect_time` VARCHAR(32) NULL COMMENT '采集时间',
  `collect_hour` VARCHAR(32) NOT NULL COMMENT '采集小时',
  `collect_minute` VARCHAR(32) NOT NULL COMMENT '采集分钟',
  `room_status_code` VARCHAR(64) NULL COMMENT '站点允许状态',
  `single_id` VARCHAR(64) NULL COMMENT '物联点',
  `is_miss` VARCHAR(32) NULL
) ENGINE=OLAP
DUPLICATE KEY(`pk_id`, `collect_date`)
COMMENT 'OLAP'
PARTITION BY RANGE(`collect_date`)()
DISTRIBUTED BY HASH(`pk_id`) BUCKETS AUTO
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p"
);

物化视图SQL

CREATE materialized VIEW mv_item_tag AS SELECT
collect_date,
room_id,
item_code,
target_code,
collect_ts,
MAX_BY ( tag_value, create_ts ) 
FROM
	test 
GROUP BY
	collect_date,
	room_id,
	item_code,
	target_code,
	collect_ts
zhangjj77