2025-03-12 20:48:40,204 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Co-Process-Broadcast -> (Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer) (1/3) (69b198b90a595dd1997944e1640dcbed_36d16ce3a2fa20764269e21927c9731c_0_2) switched from RUNNING to FAILED on flink-ess-kafka-to-doris-out-job-taskmanager-1-3 @ 10.244.4.44 (dataPort=38949).
java.io.IOException: Could not perform checkpoint 55205 for operator Co-Process-Broadcast -> (Sink: Writer -> Sink: Committer, Sink: Writer -> Sink: Committer) (1/3)#2.
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1243) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierTracker.triggerCheckpointOnAligned(CheckpointBarrierTracker.java:301) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierTracker.processBarrier(CheckpointBarrierTracker.java:141) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.io.StreamMultipleInputProcessor.processInput(StreamMultipleInputProcessor.java:85) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:545) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:836) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:785) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) ~[flink-dist-1.16.3.jar:1.16.3]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.RuntimeException: Writing records to streamload failed.
at org.apache.doris.flink.sink.batch.DorisBatchWriter.checkFlushException(DorisBatchWriter.java:168) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchWriter.flush(DorisBatchWriter.java:125) ~[?:?]
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.prepareSnapshotPreBarrier(SinkWriterOperator.java:167) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.prepareSnapshotPreBarrier(RegularOperatorChain.java:89) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:334) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1286) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1274) ~[flink-dist-1.16.3.jar:1.16.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1231) ~[flink-dist-1.16.3.jar:1.16.3]
... 17 more
Caused by: org.apache.doris.flink.exception.DorisBatchLoadException: org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad.checkFlushException(DorisBatchStreamLoad.java:301) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad.doFlush(DorisBatchStreamLoad.java:244) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad.intervalFlush(DorisBatchStreamLoad.java:235) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchWriter.intervalFlush(DorisBatchWriter.java:110) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
Caused by: org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:519) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
Caused by: org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) ~[?:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
突然所有写入doris的flink都报错，但是查看FE日志，并无相关异常
taskManager:
2025-03-12 20:37:57,678 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - load Result {"status":"FAILED","msg":"errCode = 2, detailMessage = No backend load available., policy: computeNode=false | query=false | load=true | schedule=false | tags= | medium=null"}
2025-03-12 20:37:57,678 ERROR org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load error with 172.18.255.118:8030, to retry, cause by
org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-03-12 20:37:57,685 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load started for _1_dwd_ess_other_inc_d3f3989d-9cc9-4dc4-8509-d34e1d3ee87d on host 172.18.255.118:8030
2025-03-12 20:37:57,688 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - load Result {"status":"FAILED","msg":"errCode = 2, detailMessage = No backend load available., policy: computeNode=false | query=false | load=true | schedule=false | tags= | medium=null"}
2025-03-12 20:37:57,688 ERROR org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load error with 172.18.255.118:8030, to retry, cause by
org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-03-12 20:37:57,691 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load started for _1_dwd_ess_other_inc_d3f3989d-9cc9-4dc4-8509-d34e1d3ee87d on host 172.18.255.118:8030
2025-03-12 20:37:57,693 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - load Result {"status":"FAILED","msg":"errCode = 2, detailMessage = No backend load available., policy: computeNode=false | query=false | load=true | schedule=false | tags= | medium=null"}
2025-03-12 20:37:57,693 ERROR org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load error with 172.18.255.118:8030, to retry, cause by
org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-03-12 20:37:57,696 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load started for _1_dwd_ess_other_inc_d3f3989d-9cc9-4dc4-8509-d34e1d3ee87d on host 172.18.255.118:8030
2025-03-12 20:37:57,698 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - load Result {"status":"FAILED","msg":"errCode = 2, detailMessage = No backend load available., policy: computeNode=false | query=false | load=true | schedule=false | tags= | medium=null"}
2025-03-12 20:37:57,698 ERROR org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - stream load error with 172.18.255.118:8030, to retry, cause by
org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-03-12 20:37:57,700 ERROR org.apache.doris.flink.sink.batch.DorisBatchStreamLoad [] - worker running error
org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:519) ~[blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.run(DorisBatchStreamLoad.java:407) [blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.doris.flink.exception.DorisBatchLoadException: stream load error: null, see more in null
at org.apache.doris.flink.sink.batch.DorisBatchStreamLoad$LoadAsyncExecutor.load(DorisBatchStreamLoad.java:494) ~[blob_p-22c10685ef1792e08c2f6a695172f54d6183cb8d-ac6a9b9ebcabb88f6f76228ff10481be:?]
... 4 more
