Pipeline提交的作业修改column问题

Viewed 71

doris 2.1.4
flink 1.19.1
flink-cdc 3.1.1
flink-cdc-pipeline-connector-doris-3.1.1

使用Pipeline管道提交的Yaml作业,在运行中执行了column修改alter table user change state states smallint not null default 0出现了问题,导致后面的数据都无法同步。

yaml sink配置如下

sink:
  type: doris
  fenodes: 0.0.0.0:8030
  username: *****
  password: *****
  sink.properties.format: json
  sink.properties.read_json_by_line: true
  #同步删除事件
  sink.enable-delete: true
  table.create.properties.light_schema_change: true

flink中错误如下:

Caused by: java.lang.RuntimeException: Failed to schema change, AlterColumnTypeEvent{tableId=mid_cloud_bz.user, nameMapping={state=SMALLINT NOT NULL}}, reason: Unsupported schema change event, AlterColumnTypeEvent{tableId=mid_cloud_bz.user, nameMapping={state=SMALLINT NOT NULL}}
	at org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.applySchemaChange(DorisMetadataApplier.java:87) ~[?:?]
	at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.applySchemaChange(SchemaRegistryRequestHandler.java:108) ~[?:?]
	at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.lambda$flushSuccess$0(SchemaRegistryRequestHandler.java:196) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_321]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_321]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_321]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_321]

麻烦帮忙看看这个问题, 如何恢复同步作业不影响后续数据的同步?

1 Answers

这个schema change 语句是在doris 中跑的是吧,跑完后同步任务这边就不正常了?


帖子长时间未更新,先close