集群有8个be节点，当一个be节点挂了后，flink写入出现commit异常，flink的taskmanager进入重启后尝试后，依然不能恢复。失败后重新运行，依然会出现相同问题
堆栈如下：
org.apache.flink.streaming.runtime.tasks.AsynchronousException: Caught exception while processing timer.
at org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1580) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1556) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1695) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1684) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
Caused by: org.apache.flink.streaming.runtime.tasks.TimerException: org.apache.doris.flink.exception.DorisRuntimeException: stream load error: null
... 14 more
Caused by: org.apache.doris.flink.exception.DorisRuntimeException: stream load error: null
at org.apache.doris.flink.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:108) ~[?:?]
at org.apache.doris.flink.sink.committer.DorisCommitter.commit(DorisCommitter.java:71) ~[?:?]
at org.apache.flink.streaming.runtime.operators.sink.StreamingCommitterHandler.commit(StreamingCommitterHandler.java:54) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.AbstractStreamingCommitterHandler.retry(AbstractStreamingCommitterHandler.java:99) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.AbstractCommitterHandler.retry(AbstractCommitterHandler.java:66) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.retry(CommitRetrier.java:80) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.lambda$retryAt$0(CommitRetrier.java:63) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1693) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
... 13 more
更详细的报错：
2024-10-12 14:13:47,461 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka version: 2.4.1
2024-10-12 14:13:47,461 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka commitId: c57222ae8cd7866b
2024-10-12 14:13:47,461 INFO org.apache.kafka.common.utils.AppInfoParser [] - Kafka startTimeMs: 1728713627460
2024-10-12 14:13:47,477 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Starting split fetcher 0
2024-10-12 14:13:47,549 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: Kafka Source -> Flat Map
-> Flat Map -> Flat Map -> Sink Unnamed (35/50)#0 (6d65f136e82f5e13509865204c8ff207) switched from INITIALIZING to RUNNING.
2024-10-12 14:13:47,559 INFO org.apache.kafka.clients.consumer.KafkaConsumer [] - [Consumer clientId=gateway_test_d
oris0007-34, groupId=gateway_test_doris0007] Subscribed to partition(s): gatewayaccesslog-38
2024-10-12 14:13:47,575 INFO org.apache.kafka.clients.consumer.KafkaConsumer [] - [Consumer clientId=gateway_test_d
oris0007-34, groupId=gateway_test_doris0007] Seeking to offset 58940894102 for partition gatewayaccesslog-38
2024-10-12 14:13:48,049 ERROR org.apache.doris.flink.sink.committer.DorisCommitter [] - commit transaction failed:
org.apache.http.conn.HttpHostConnectException: Connect to 10.26.72.64:8040 [/10.26.72.64] failed: Connection refused (Connection
refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[
blob_p-61f0320fe7cc111cc844c6575ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[bl
ob_p-61f0320fe7cc111cc844c6575ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[blob_p-61f0320fe7cc111cc844c65
75ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[blob_p-61f0320fe7cc111cc844c6575ef7cc
48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[blob_p-61f0320fe7cc111cc844c6575ef7cc48af
8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[blob_p-61f0320fe7cc111cc844c6575ef7cc48af8a4b1f-
139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[blob_p-61f0320fe7cc111cc844c6575ef7cc48af
8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[blob_p-61f0320fe7cc111cc844c65
75ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[blob_p-61f0320fe7cc111cc844c657
5ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[blob_p-61f0320fe7cc111cc844c65
75ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.doris.flink.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:91) ~[blob_p-61f0320fe7cc11
1cc844c6575ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.doris.flink.sink.committer.DorisCommitter.commit(DorisCommitter.java:71) ~[blob_p-61f0320fe7cc111cc844c6575
ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.flink.streaming.runtime.operators.sink.StreamingCommitterHandler.commit(StreamingCommitterHandler.java:54)
~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.AbstractStreamingCommitterHandler.retry(AbstractStreamingCommitterHa
ndler.java:99) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.AbstractCommitterHandler.retry(AbstractCommitterHandler.java:66) ~[f
link-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.retry(CommitRetrier.java:80) ~[flink-dist_2.11-1.14.4.
jar:1.14.4]
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.lambda$retryAt$0(CommitRetrier.java:63) ~[flink-dist_2
.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1693) ~[flink-dist_2.
11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1684) ~[flink-dist_2.11-1.14.4.jar:
1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[fl
ink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProc
essor.java:338) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) ~[flink-dist_
2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-di
st_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.11-1.14.4.jar:1
.14.4]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) [flink-dist_2.11-1.14.4.jar:1.14.
4]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) [flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) [flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist_2.11-1.14.4.jar:1.14.4]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_322]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_322]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_322]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[blob_p-
61f0320fe7cc111cc844c6575ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[
blob_p-61f0320fe7cc111cc844c6575ef7cc48af8a4b1f-139883e44a5481fda2030ca9306e6ec8:?]
... 30 more
2024-10-12 14:13:48,065 INFO org.apache.doris.flink.sink.committer.DorisCommitter [] - Send request to Doris FE 'http://
10.26.72.58:8030/api/backends?is_alive=true' with user 'tuser_rw'.
2024-10-12 14:13:48,148 INFO org.apache.doris.flink.sink.committer.DorisCommitter [] - Backend Info:{"backends":[{"ip":"
10.26.72.65","http_port":8040,"is_alive":true},{"ip":"10.26.72.66","http_port":8040,"is_alive":true},{"ip":"10.26.72.61","http_po
rt":8040,"is_alive":true},{"ip":"10.26.72.67","http_port":8040,"is_alive":true},{"ip":"10.26.72.68","http_port":8040,"is_alive":t
rue},{"ip":"10.26.72.62","http_port":8040,"is_alive":true},{"ip":"10.26.72.63","http_port":8040,"is_alive":true}]}
2024-10-12 14:13:48,153 INFO org.apache.flink.runtime.taskmanager.Task [] - Attempting to fail task externall
y Source: Kafka Source -> Flat Map -> Flat Map -> Flat Map -> Sink Unnamed (35/50)#0 (6d65f136e82f5e13509865204c8ff207).
2024-10-12 14:13:48,155 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: Kafka Source -> Flat Map
-> Flat Map -> Flat Map -> Sink Unnamed (35/50)#0 (6d65f136e82f5e13509865204c8ff207) switched from RUNNING to FAILED with failure
cause: org.apache.flink.streaming.runtime.tasks.AsynchronousException: Caught exception while processing timer.
at org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.ja
va:1580)
at org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1556)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1695)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1684)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProc
essor.java:338)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:750)
Caused by: TimerException{org.apache.doris.flink.exception.DorisRuntimeException: stream load error: null}
... 14 more
Caused by: org.apache.doris.flink.exception.DorisRuntimeException: stream load error: null
at org.apache.doris.flink.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:108)
at org.apache.doris.flink.sink.committer.DorisCommitter.commit(DorisCommitter.java:71)
at org.apache.flink.streaming.runtime.operators.sink.StreamingCommitterHandler.commit(StreamingCommitterHandler.java:54)
at org.apache.flink.streaming.runtime.operators.sink.AbstractStreamingCommitterHandler.retry(AbstractStreamingCommitterHa
ndler.java:99)
at org.apache.flink.streaming.runtime.operators.sink.AbstractCommitterHandler.retry(AbstractCommitterHandler.java:66)
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.retry(CommitRetrier.java:80)
at org.apache.flink.streaming.runtime.operators.sink.CommitRetrier.lambda$retryAt$0(CommitRetrier.java:63)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1693)
... 13 more
2024-10-12 14:13:48,156 INFO org.apache.flink.runtime.taskmanager.Task [] - Triggering cancellation of task c
ode Source: Kafka Source -> Flat Map -> Flat Map -> Flat Map -> Sink Unnamed (35/50)#0 (6d65f136e82f5e13509865204c8ff207).
2024-10-12 14:13:48,253 INFO org.apache.kafka.clients.producer.KafkaProducer [] - [Producer clientId=producer-1] Cl
osing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2024-10-12 14:13:48,265 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Closing Source Reader.
2024-10-12 14:13:48,266 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Shutting down split fetcher
0
2024-10-12 14:13:48,275 INFO org.apache.kafka.clients.Metadata [] - [Consumer clientId=gateway_test_d
oris0007-34, groupId=gateway_test_doris0007] Cluster ID: N5fjoUD3TiyIEVhrJg7waQ
2024-10-12 14:13:48,355 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Split fetcher 0 exited.
2024-10-12 14:13:48,360 INFO org.apache.flink.runtime.taskmanager.Task [] - Freeing task resources for Source
: Kafka Source -> Flat Map -> Flat Map -> Flat Map -> Sink Unnamed (35/50)#0 (6d65f136e82f5e13509865204c8ff207).
2024-10-12 14:13:48,371 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Un-registering task and sending f
inal execution state FAILED to JobManager for task Source: Kafka Source -> Flat Map -> Flat Map -> Flat Map -> Sink Unnamed (35/5
0)#0 6d65f136e82f5e13509865204c8ff207.
2024-10-12 14:15:13,067 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot TaskSlot(index:0, state
:ACTIVE, resource profile: ResourceProfile{cpuCores=1, taskHeapMemory=1.425gb (1530082070 bytes), taskOffHeapMemory=0 bytes, mana
gedMemory=1.340gb (1438814063 bytes), networkMemory=343.040mb (359703515 bytes)}, allocationId: d6390820383e8c94ee6dbbfab22c4fa8,
jobId: 5db4dd6f877ace6fc8fb038ecc836a96).
2024-10-12 14:15:13,071 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Remove job 5db4dd6f877ace6fc8fb0
38ecc836a96 from job leader monitoring.
2024-10-12 14:15:13,072 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Stopping DefaultLeaderR
etrievalService.
2024-10-12 14:15:13,072 INFO org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver [] - Stopping Kubernet
esLeaderRetrievalDriver{configMapName='fm-hv7j6qxufi-5db4dd6f877ace6fc8fb038ecc836a96-jobmanager-leader'}.
2024-10-12 14:15:13,072 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Stopped to
watch for ark-middleware-flink-0/fm-hv7j6qxufi-5db4dd6f877ace6fc8fb038ecc836a96-jobmanager-leader, watching id:d0ebe9c1-4861-41ab
-90cd-99c679460b65
2024-10-12 14:15:13,073 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close JobManager connection for j
ob 5db4dd6f877ace6fc8fb038ecc836a96.
2024-10-12 14:15:51,686 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close ResourceManager connection
f5a7997337109e2f65398e018673628c.
2024-10-12 14:15:51,691 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Connecting to ResourceManager akk
a.tcp://flink@10.26.129.183:6123/user/rpc/resourcemanager_1(9f459ca8fa63a2245aee375cf21644df).
2024-10-12 14:15:51,701 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Resolved ResourceManager address,
beginning registration
2024-10-12 14:15:51,747 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Fatal error occurred in TaskExecu
tor akka.tcp://flink@10.26.130.154:6122/user/rpc/taskmanager_0.
org.apache.flink.util.FlinkException: The TaskExecutor's registration at the ResourceManager akka.tcp://flink@10.26.129.183:6123/
user/rpc/resourcemanager_1 has been rejected: Rejected TaskExecutor registration at the ResourceManager because: The ResourceMana
ger does not recognize this TaskExecutor.
at org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerRegistrationListener.onRegistrationRejection(TaskExe
cutor.java:2293) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerRegistrationListener.onRegistrationRejection(TaskExe
cutor.java:2248) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutorToResourceManagerConnection.onRegistrationRejection(TaskExecutorToRe
sourceManagerConnection.java:109) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutorToResourceManagerConnection.onRegistrationRejection(TaskExecutorToRe
sourceManagerConnection.java:40) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.registration.RegisteredRpcConnection.lambda$createNewRegistration$0(RegisteredRpcConnection.j
ava:269) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_322]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455) ~[flink-rpc-akka_61554e9
8-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flin
k-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455) ~[flink-rpc-akka_61554e98-82ef-4b
21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[flink-rpc-akka_61554e98-82ef-
4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163) ~[flink-rpc-akka_61554e98-82ef-4b2
1-aa04-410f34b3bd55.jar:1.14.4]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:
1.14.4]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.j
ar:1.14.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14
.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.
4]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_322]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_322]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_322]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_322]
2024-10-12 14:15:51,749 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Fatal error occurred while execut
ing the TaskManager. Shutting it down...
org.apache.flink.util.FlinkException: The TaskExecutor's registration at the ResourceManager akka.tcp://flink@10.26.129.183:6123/
user/rpc/resourcemanager_1 has been rejected: Rejected TaskExecutor registration at the ResourceManager because: The ResourceMana
ger does not recognize this TaskExecutor.
at org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerRegistrationListener.onRegistrationRejection(TaskExe
cutor.java:2293) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerRegistrationListener.onRegistrationRejection(TaskExe
cutor.java:2248) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutorToResourceManagerConnection.onRegistrationRejection(TaskExecutorToRe
sourceManagerConnection.java:109) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.taskexecutor.TaskExecutorToResourceManagerConnection.onRegistrationRejection(TaskExecutorToRe
sourceManagerConnection.java:40) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at org.apache.flink.runtime.registration.RegisteredRpcConnection.lambda$createNewRegistration$0(RegisteredRpcConnection.j
ava:269) ~[flink-dist_2.11-1.14.4.jar:1.14.4]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_322]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455) ~[flink-rpc-akka_61554e9
8-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flin
k-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455) ~[flink-rpc-akka_61554e98-82ef-4b
21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[flink-rpc-akka_61554e98-82ef-
4b21-aa04-410f34b3bd55.jar:1.14.4]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163) ~[flink-rpc-akka_61554e98-82ef-4b2
1-aa04-410f34b3bd55.jar:1.14.4]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:
1.14.4]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar
:1.14.4]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd
55.jar:1.14.4]
at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.j
ar:1.14.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14
.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.
4]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_61554e98-82ef-4b21-aa04-410f34b3bd55.jar:1.14.4]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_322]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_322]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_322]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_322]
2024-10-12 14:15:51,758 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Stopping TaskExecutor akka.tcp://
flink@10.26.130.154:6122/user/rpc/taskmanager_0.
2024-10-12 14:15:51,758 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Terminating registration attempts
towards ResourceManager akka.tcp://flink@10.26.129.183:6123/user/rpc/resourcemanager_1.
2024-10-12 14:15:51,758 INFO org.apache.flink.runtime.state.TaskExecutorStateChangelogStoragesManager [] - Shutting down TaskExe
cutorStateChangelogStoragesManager.
2024-10-12 14:15:51,763 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Stop job leader service.
2024-10-12 14:15:51,763 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Stopping DefaultLeaderR
etrievalService.
2024-10-12 14:15:51,763 INFO org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver [] - Stopping Kubernet
esLeaderRetrievalDriver{configMapName='fm-hv7j6qxufi-resourcemanager-leader'}.
2024-10-12 14:15:51,763 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Shutting down TaskExecutorL
ocalStateStoresManager.
2024-10-12 14:15:51,763 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer [] - Stopped to
watch for ark-middleware-flink-0/fm-hv7j6qxufi-resourcemanager-leader, watching id:38289e44-29df-4397-b6f4-8695ba7336d4
2024-10-12 14:15:51,767 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl [] - FileChannelManager removed spill
file directory /tmp/flink-io-430084b7-2019-48a1-910f-73d9ad8d853e
2024-10-12 14:15:51,768 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment [] - Shutting down the network environ
ment and its components.
2024-10-12 14:15:51,768 INFO org.apache.flink.runtime.io.network.netty.NettyClient [] - Successful shutdown (took 0 ms).
2024-10-12 14:15:51,770 INFO org.apache.flink.runtime.io.network.netty.NettyServer [] - Successful shutdown (took 1 ms).
2024-10-12 14:15:51,774 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - RECEIVED SIGNAL 15: SIGTE
RM. Shutting down as requested.
2024-10-12 14:15:51,775 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl [] - FileChannelManager removed spill
file directory /tmp/flink-netty-shuffle-b299a9a1-9954-4ee0-b1fc-2408d4186c7b
2024-10-12 14:15:51,776 INFO org.apache.flink.runtime.taskexecutor.KvStateService [] - Shutting down the kvState service
and its components.
2024-10-12 14:15:51,777 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Stop job leader service.
2024-10-12 14:15:51,777 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting down BLOB cache
2024-10-12 14:15:51,778 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting down BLOB cache
2024-10-12 14:15:51,778 INFO org.apache.flink.runtime.filecache.FileCache [] - removed file cache directory /tmp
/flink-dist-cache-ac572773-4395-4ac7-8fb9-0a87e4d18bb0
2024-10-12 14:15:51,781 INFO org.apache.flink.runtime.filecache.FileCache [] - removed file cache directory /tmp
/flink-dist-cache-ac572773-4395-4ac7-8fb9-0a87e4d18bb0
2024-10-12 14:15:51,790 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Stopped TaskExecutor akka.tcp://f
link@10.26.130.154:6122/user/rpc/taskmanager_0.
