【已解决】flink-doris-connectors 消费kafka 写入报错：TStatus: errCode = 2, detailMessage = Unknown column 'OCCURTIME' in 't1', see more in null

Viewed 47
2.1

1.doris版本 2.1.3，flink-doris-connectors:1.6.1; flink 1.17.0
2.场景：flink-doris-connectors 消费kafka 写入doris,前期正常写入，重启任务后报错，错误如下：

java.lang.Exception: Could not perform checkpoint 50 for operator Source: db-tolldata-prod-1 -> Process -> (Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer) (1/3)#47.
	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointAsyncInMailbox(StreamTask.java:1184)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$triggerCheckpointAsync$13(StreamTask.java:1131)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.doris.flink.exception.DorisRuntimeException: table ods_tolldata_prod.ods_t_otherexit_waste stream load error: [ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Unknown column 'OCCURTIME' in 't1', see more in null
	at org.apache.doris.flink.sink.writer.DorisWriter.prepareCommit(DorisWriter.java:274)

检查table ods_tolldata_prod.ods_t_otherexit_waste，schema信息无误。
1.尝试解决：重新建表后可以写入。但是历史数据已经写进去大量数据，同步表数200张，重新建表很耗时，也会丢失历史已经同步的数据；

hopet5
2 Answers

这个是老版本的Auto Partition功能的一个bug，我们在2.1.4已经修复了这个问题。这主要是由于集群压力过大，FE回退到老优化器，触发了老优化器的一些bug导致的。在2.1.4之前的版本，可以手动关闭回退来规避这个问题：

set global enable_fallback_to_original_planner=false;
zclllyybb361

【问题状态】处理中

张彬华5610