flink 的job从checkpoints恢复报错,使用flink-doris-connector-1.17-1.6.1.jar报错,使用flink-doris-connector-1.17-1.4.0.jar正常
insert into sql的job从checkpoints恢复正常,但是直接用
DorisSink sink
stream.map(e -> JSON.toJSONString(e)).sinkTo(sink)
这种写法的job还是会有这个错误,
之前1.6.0的jar包在insert into 的flink sql的job从checkpoints是这种也会报错,现在1.6.1修复了,但是datastream里面用DorisSink写的情况还是报错
2024-06-18 10:41:21,660 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - xxx -> Map -> doris-sink-xxx: Writer -> doris-sink-xxx: Committer (2/4) (40e201018ac18ff1ec9968b3236d0df1_f97b0d875bf3f5b84d23affac6dc08ec_1_0) switched from INITIALIZING to FAILED on container_e12_1713251823360_0253_01_000002 @ cc-gzgz1dev-x86-ceph-1 (dataPort=43710).
org.apache.doris.flink.exception.DorisRuntimeException: commit transaction error,
at org.apache.doris.flink.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:162) ~[flink-doris-connector-1.17-1.6.1.jar:1.6.1]
at org.apache.doris.flink.sink.committer.DorisCommitter.commit(DorisCommitter.java:94) ~[flink-doris-connector-1.17-1.6.1.jar:1.6.1]
at org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-dist-1.17.2.jar:1.17.2]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: org.apache.doris.flink.exception.DorisRuntimeException: commit transaction failed {
"status": "INTERNAL_ERROR",
"msg": "TStatus: "
}
at org.apache.doris.flink.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:129) ~[flink-doris-connector-1.17-1.6.1.jar:1.6.1]
... 17 more
doris的fe报错
2024-06-18 10:41:21,321 WARN (thrift-server-pool-8|495) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,321 WARN (thrift-server-pool-16|3550) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,321 WARN (thrift-server-pool-17|3551) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,321 WARN (thrift-server-pool-20|3554) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,392 INFO (qtp554577711-4246768|4246768) [LoadAction.executeStreamLoad2PC():220] redirect stream load 2PC action to destination=TNetworkAddress(hostname:xxx.xxx.xxx.xxx, port:xxxx), db: cloud_desktop, txn: 42204152, operation: commit
2024-06-18 10:41:21,392 INFO (qtp554577711-4246768|4246768) [RestBaseController.redirectTo():109] Redirect url: http://xxx.xxx.xxx.xxx:xxxx/api/cloud_desktop/_stream_load_2pc
2024-06-18 10:41:21,392 INFO (qtp554577711-4249057|4249057) [LoadAction.executeStreamLoad2PC():220] redirect stream load 2PC action to destination=TNetworkAddress(hostname:xxx.xxx.xxx.xxx, port:xxxx), db: cloud_desktop, txn: 42204151, operation: commit
2024-06-18 10:41:21,393 INFO (qtp554577711-4249057|4249057) [RestBaseController.redirectTo():109] Redirect url: http://xxx.xxx.xxx.xxx:xxxx/api/cloud_desktop/_stream_load_2pc
2024-06-18 10:41:21,393 INFO (qtp554577711-4260671|4260671) [LoadAction.executeStreamLoad2PC():220] redirect stream load 2PC action to destination=TNetworkAddress(hostname:xxx.xxx.xxx.xxx, port:xxxx), db: cloud_desktop, txn: 42204162, operation: commit
2024-06-18 10:41:21,393 INFO (qtp554577711-4260671|4260671) [RestBaseController.redirectTo():109] Redirect url: http://xxx.xxx.xxx.xxx:xxxx/api/cloud_desktop/_stream_load_2pc
2024-06-18 10:41:21,393 INFO (qtp554577711-4232903|4232903) [LoadAction.executeStreamLoad2PC():220] redirect stream load 2PC action to destination=TNetworkAddress(hostname:xxx.xxx.xxx.xxx, port:xxxx), db: cloud_desktop, txn: 42204163, operation: commit
2024-06-18 10:41:21,393 INFO (qtp554577711-4232903|4232903) [RestBaseController.redirectTo():109] Redirect url: http://xxx.xxx.xxx.xxx:xxxx/api/cloud_desktop/_stream_load_2pc
2024-06-18 10:41:21,601 WARN (thrift-server-pool-139|1449870) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,601 WARN (thrift-server-pool-222|2204587) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,601 WARN (thrift-server-pool-13|3487) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:21,601 WARN (thrift-server-pool-223|2204588) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:22,845 WARN (thrift-server-pool-18|3552) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.
2024-06-18 10:41:22,847 WARN (thrift-server-pool-27|3566) [FrontendServiceImpl.loadTxn2PC():1494] catch unknown result.