需求：doris表有3个字段,id(int),age(int),name(string),采用主键模型,实现部分字段更新
下面以sql方式为例：api的也是一样情况
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
<version>24.0.0</version>
</dependency>
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
String sink = "CREATE TABLE sink (" +
" id INT," +
" age INT" +
") " +
" WITH (" +
" 'connector' = 'doris'," +
" 'fenodes' = 'xxx:8030'," +
" 'table.identifier' = 'test_model.test_primary3'," +
" 'username' = 'root'," +
" 'password' = ''," +
" 'sink.label-prefix' = 'zzz_label'," +
" 'sink.enable.batch-mode' = 'true'," +
" 'sink.buffer-flush.max-rows' = '100000'," +
" 'sink.buffer-flush.max-bytes' = '10485760000'," +
" 'sink.buffer-flush.interval' = '30000'," +
" 'sink.properties.column' = 'id,age'," +
" 'sink.properties.partial_columns' = 'true'" +
");";
String sql = "insert into sink(id,age) values(1,19)";
tableEnv.executeSql(sink);
tableEnv.executeSql(sql);
3个参数设置后，依旧是任务执行，立马插入，日志没有明显的看出来有阻塞的情况。使用的是本地idea跑的，下面是日志打印：
2024-09-25 22:38:42,104 INFO org.apache.doris.flink.table.DorisDynamicTableSink - Send request to Doris FE 'http://xxxx/api/test_model/test_primary3/_schema' with user 'root'.
2024-09-25 22:38:42,346 INFO org.apache.doris.flink.sink.DorisSink - Send request to Doris FE 'http://xxxx/api/test_model/test_primary3/_schema' with user 'root'.
2024-09-25 22:38:42,563 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.cpu.cores required for local execution is not set, setting it to the maximal possible value.
2024-09-25 22:38:42,563 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.task.heap.size required for local execution is not set, setting it to the maximal possible value.
2024-09-25 22:38:42,563 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.task.off-heap.size required for local execution is not set, setting it to the maximal possible value.
2024-09-25 22:38:42,563 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.network.min required for local execution is not set, setting it to its default value 64 mb.
2024-09-25 22:38:42,564 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.network.max required for local execution is not set, setting it to its default value 64 mb.
2024-09-25 22:38:42,564 INFO org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils - The configuration option taskmanager.memory.managed.size required for local execution is not set, setting it to its default value 128 mb.
2024-09-25 22:38:42,568 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Flink Mini Cluster
2024-09-25 22:38:42,857 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting Metrics Registry
2024-09-25 22:38:42,911 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2024-09-25 22:38:42,911 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting RPC Service(s)
2024-09-25 22:38:42,925 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start local actor system
2024-09-25 22:38:43,482 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2024-09-25 22:38:43,571 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka://flink
2024-09-25 22:38:43,583 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start local actor system
2024-09-25 22:38:43,591 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2024-09-25 22:38:43,599 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka://flink-metrics
2024-09-25 22:38:43,609 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
2024-09-25 22:38:43,637 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory C:\Users\zz\AppData\Local\Temp\minicluster_f646a7c1e4a26d512f0e63eeeb235219\blobStorage
2024-09-25 22:38:43,640 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:4723 - max concurrent requests: 50 - max backlog: 1000
2024-09-25 22:38:43,645 INFO org.apache.flink.runtime.security.token.KerberosDelegationTokenManagerFactory - Cannot use kerberos delegation token manager because Hadoop cannot be found in the Classpath.
2024-09-25 22:38:43,649 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory C:\Users\zz\AppData\Local\Temp\minicluster_f646a7c1e4a26d512f0e63eeeb235219\blobStorage
2024-09-25 22:38:43,651 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory C:\Users\zz\AppData\Local\Temp\minicluster_f646a7c1e4a26d512f0e63eeeb235219\blobStorage
2024-09-25 22:38:43,652 INFO org.apache.flink.runtime.minicluster.MiniCluster - Starting 1 TaskManager(s)
2024-09-25 22:38:43,658 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager with ResourceID: 4eb7c128-3895-4260-8d21-ebaf9dd3ac92
2024-09-25 22:38:43,672 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory 'C:\Users\zz\AppData\Local\Temp': total 152 GB, usable 20 GB (13.16% usable)
2024-09-25 22:38:43,680 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - Created a new FileChannelManager for spilling of task related data to disk (joins, sorting, ...). Used directories:
C:\Users\zz\AppData\Local\Temp\flink-io-9bcf4fd8-0a9f-4b22-a7f8-0d6239767bf9
2024-09-25 22:38:43,693 INFO org.apache.flink.runtime.io.network.NettyShuffleServiceFactory - Created a new FileChannelManager for storing result partitions of BLOCKING shuffles. Used directories:
C:\Users\zz\AppData\Local\Temp\flink-netty-shuffle-55a4a9d0-2286-4196-a134-e25efbac28dd
2024-09-25 22:38:43,731 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
2024-09-25 22:38:43,755 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Starting the network environment and its components.
2024-09-25 22:38:43,758 INFO org.apache.flink.runtime.taskexecutor.KvStateService - Starting the kvState service and its components.
2024-09-25 22:38:43,779 INFO org.apache.flink.configuration.Configuration - Config uses fallback configuration key 'akka.ask.timeout' instead of key 'taskmanager.slot.timeout'
2024-09-25 22:38:43,799 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 .
2024-09-25 22:38:43,816 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Start job leader service.
2024-09-25 22:38:43,819 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory C:\Users\zz\AppData\Local\Temp\flink-dist-cache-43c9b1e7-dd62-42f5-899e-ef0cd563f112
2024-09-25 22:38:43,855 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2024-09-25 22:38:44,029 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2024-09-25 22:38:44,029 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'.
2024-09-25 22:38:44,665 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at localhost:4790
2024-09-25 22:38:44,665 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender http://localhost:4790
2024-09-25 22:38:44,667 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://localhost:4790.
2024-09-25 22:38:44,667 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://localhost:4790 was granted leadership with leaderSessionID=de9a485c-c4ce-4fc8-8633-cd1f22188b84
2024-09-25 22:38:44,667 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader http://localhost:4790 , session=de9a485c-c4ce-4fc8-8633-cd1f22188b84
2024-09-25 22:38:44,682 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender LeaderContender: DefaultDispatcherRunner
2024-09-25 22:38:44,682 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl - Starting resource manager service.
2024-09-25 22:38:44,683 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender LeaderContender: ResourceManagerServiceImpl
2024-09-25 22:38:44,683 INFO org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner - DefaultDispatcherRunner was granted leadership with leader id ff086ea0-880a-44ff-ac70-63f17ee3cd82. Creating new DispatcherLeaderProcess.
2024-09-25 22:38:44,684 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl - Resource manager service is granted leadership with session id 270f650b-882b-4bf3-abae-5dff5d1922b4.
2024-09-25 22:38:44,689 INFO org.apache.flink.runtime.minicluster.MiniCluster - Flink Mini Cluster started successfully
2024-09-25 22:38:44,692 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Start SessionDispatcherLeaderProcess.
2024-09-25 22:38:44,695 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Recover all persisted job graphs that are not finished, yet.
2024-09-25 22:38:44,695 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Successfully recovered 0 persisted job graphs.
2024-09-25 22:38:44,711 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_1 .
2024-09-25 22:38:44,712 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_2 .
2024-09-25 22:38:44,723 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Starting the resource manager.
2024-09-25 22:38:44,728 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/dispatcher_2 , session=ff086ea0-880a-44ff-ac70-63f17ee3cd82
2024-09-25 22:38:44,729 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/resourcemanager_1 , session=270f650b-882b-4bf3-abae-5dff5d1922b4
2024-09-25 22:38:44,766 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(abae5dff5d1922b4270f650b882b4bf3).
2024-09-25 22:38:44,771 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration
2024-09-25 22:38:44,774 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering TaskManager with ResourceID 4eb7c128-3895-4260-8d21-ebaf9dd3ac92 (akka://flink/user/rpc/taskmanager_0) at ResourceManager
2024-09-25 22:38:44,776 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received JobGraph submission 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:44,776 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:44,776 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka://flink/user/rpc/resourcemanager_1 under registration id be2c8a5712668ecc0ef4a45e263439ec.
2024-09-25 22:38:44,791 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Proposing leadership to contender LeaderContender: JobMasterServiceLeadershipRunner
2024-09-25 22:38:44,805 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 .
2024-09-25 22:38:44,814 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:44,840 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart back off time strategy NoRestartBackoffTimeStrategy for insert-into_default_catalog.default_database.sink (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:44,879 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Created execution graph a364a10472f75b65b54f92b065f3617c for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:44,901 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job insert-into_default_catalog.default_database.sink (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:44,906 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 5 ms.
2024-09-25 22:38:44,942 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology - Built 1 new pipelined regions in 2 ms, total 1 pipelined regions currently.
2024-09-25 22:38:44,956 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@4a6d04b9
2024-09-25 22:38:44,956 INFO org.apache.flink.runtime.state.StateBackendLoader - State backend loader loads the state backend as HashMapStateBackend
2024-09-25 22:38:44,958 INFO org.apache.flink.runtime.jobmaster.JobMaster - Checkpoint storage is set to 'jobmanager'
2024-09-25 22:38:45,006 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No checkpoint found during restore.
2024-09-25 22:38:45,017 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@1501ed46 for insert-into_default_catalog.default_database.sink (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:45,027 INFO org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Received confirmation of leadership for leader akka://flink/user/rpc/jobmanager_3 , session=1dc36010-c42d-48dd-a09b-7b2756449339
2024-09-25 22:38:45,030 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae) under job master id a09b7b27564493391dc36010c42d48dd.
2024-09-25 22:38:45,032 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy]
2024-09-25 22:38:45,032 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job insert-into_default_catalog.default_database.sink (eee7e6b18a8ea0778e05d68efd0520ae) switched from state CREATED to RUNNING.
2024-09-25 22:38:45,034 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Values[1] (1/1) (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from CREATED to SCHEDULED.
2024-09-25 22:38:45,035 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - sink[2]: Writer -> sink[2]: Committer (1/1) (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from CREATED to SCHEDULED.
2024-09-25 22:38:45,049 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(abae5dff5d1922b4270f650b882b4bf3)
2024-09-25 22:38:45,051 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration
2024-09-25 22:38:45,052 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering job manager a09b7b27564493391dc36010c42d48dd@akka://flink/user/rpc/jobmanager_3 for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:45,057 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registered job manager a09b7b27564493391dc36010c42d48dd@akka://flink/user/rpc/jobmanager_3 for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:45,058 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: abae5dff5d1922b4270f650b882b4bf3.
2024-09-25 22:38:45,060 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager - Received resource requirements from job eee7e6b18a8ea0778e05d68efd0520ae: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2024-09-25 22:38:45,134 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot request fd7235e8279573693eed5b6e8f0c866d for job eee7e6b18a8ea0778e05d68efd0520ae from resource manager with leader id abae5dff5d1922b4270f650b882b4bf3.
2024-09-25 22:38:45,145 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot for fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,147 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Add job eee7e6b18a8ea0778e05d68efd0520ae for job leader monitoring.
2024-09-25 22:38:45,150 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Try to register at job manager akka://flink/user/rpc/jobmanager_3 with leader id 1dc36010-c42d-48dd-a09b-7b2756449339.
2024-09-25 22:38:45,151 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Resolved JobManager address, beginning registration
2024-09-25 22:38:45,158 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Successful registration at job manager akka://flink/user/rpc/jobmanager_3 for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:45,158 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish JobManager connection for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:45,161 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved slots to the leader of job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:45,170 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Values[1] (1/1) (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from SCHEDULED to DEPLOYING.
2024-09-25 22:38:45,179 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Values[1] (1/1) (attempt #0) with attempt id a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0 and vertex id bc764cd8ddf7a0cff126f51c16239658_0 to 4eb7c128-3895-4260-8d21-ebaf9dd3ac92 @ 127.0.0.1 (dataPort=-1) with allocation id fd7235e8279573693eed5b6e8f0c866d
2024-09-25 22:38:45,186 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - sink[2]: Writer -> sink[2]: Committer (1/1) (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from SCHEDULED to DEPLOYING.
2024-09-25 22:38:45,187 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Activate slot fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,187 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying sink[2]: Writer -> sink[2]: Committer (1/1) (attempt #0) with attempt id a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0 and vertex id 20ba6b65f97481d5570070de90e4e791_0 to 4eb7c128-3895-4260-8d21-ebaf9dd3ac92 @ 127.0.0.1 (dataPort=-1) with allocation id fd7235e8279573693eed5b6e8f0c866d
2024-09-25 22:38:45,211 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader - StateChangelogStorageLoader initialized with shortcut names {memory}.
2024-09-25 22:38:45,212 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader - Creating a changelog storage with name 'memory'.
2024-09-25 22:38:45,239 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0), deploy into slot with allocation id fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,240 INFO org.apache.flink.runtime.taskmanager.Task - Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from CREATED to DEPLOYING.
2024-09-25 22:38:45,241 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Activate slot fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,246 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) [DEPLOYING].
2024-09-25 22:38:45,259 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0), deploy into slot with allocation id fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,260 INFO org.apache.flink.runtime.taskmanager.Task - sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from CREATED to DEPLOYING.
2024-09-25 22:38:45,260 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Activate slot fd7235e8279573693eed5b6e8f0c866d.
2024-09-25 22:38:45,261 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) [DEPLOYING].
2024-09-25 22:38:45,274 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@ee62288
2024-09-25 22:38:45,275 INFO org.apache.flink.runtime.state.StateBackendLoader - State backend loader loads the state backend as HashMapStateBackend
2024-09-25 22:38:45,275 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Checkpoint storage is set to 'jobmanager'
2024-09-25 22:38:45,276 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@31b9e89d
2024-09-25 22:38:45,276 INFO org.apache.flink.runtime.state.StateBackendLoader - State backend loader loads the state backend as HashMapStateBackend
2024-09-25 22:38:45,276 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Checkpoint storage is set to 'jobmanager'
2024-09-25 22:38:45,285 INFO org.apache.flink.runtime.taskmanager.Task - sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from DEPLOYING to INITIALIZING.
2024-09-25 22:38:45,286 INFO org.apache.flink.runtime.taskmanager.Task - Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from DEPLOYING to INITIALIZING.
2024-09-25 22:38:45,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - sink[2]: Writer -> sink[2]: Committer (1/1) (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from DEPLOYING to INITIALIZING.
2024-09-25 22:38:45,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Values[1] (1/1) (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from DEPLOYING to INITIALIZING.
2024-09-25 22:38:45,368 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation - No hostname could be resolved for the IP address 127.0.0.1, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted.
2024-09-25 22:38:45,372 INFO org.apache.flink.runtime.taskmanager.Task - Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from INITIALIZING to RUNNING.
2024-09-25 22:38:45,372 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Values[1] (1/1) (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from INITIALIZING to RUNNING.
2024-09-25 22:38:45,432 INFO org.apache.flink.runtime.taskmanager.Task - Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to FINISHED.
2024-09-25 22:38:45,432 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Values[1] (1/1)#0 (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0).
2024-09-25 22:38:45,435 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task Source: Values[1] (1/1)#0 a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0.
2024-09-25 22:38:45,452 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Values[1] (1/1) (a364a10472f75b65b54f92b065f3617c_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to FINISHED.
2024-09-25 22:38:45,971 INFO org.apache.doris.flink.sink.batch.DorisBatchWriter - labelPrefix zzz_label
2024-09-25 22:38:45,984 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad - LoadAsyncExecutor start
2024-09-25 22:38:45,987 INFO org.apache.flink.runtime.taskmanager.Task - sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from INITIALIZING to RUNNING.
2024-09-25 22:38:45,988 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - sink[2]: Writer -> sink[2]: Committer (1/1) (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from INITIALIZING to RUNNING.
2024-09-25 22:38:46,002 INFO org.apache.doris.flink.sink.batch.DorisBatchWriter - checkpoint flush triggered.
2024-09-25 22:38:46,007 INFO org.apache.doris.flink.sink.batch.DorisBatchWriter - DorisBatchWriter Close
2024-09-25 22:38:46,008 INFO org.apache.flink.runtime.taskmanager.Task - sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from RUNNING to FINISHED.
2024-09-25 22:38:46,008 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for sink[2]: Writer -> sink[2]: Committer (1/1)#0 (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0).
2024-09-25 22:38:46,009 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task sink[2]: Writer -> sink[2]: Committer (1/1)#0 a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0.
2024-09-25 22:38:46,010 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - sink[2]: Writer -> sink[2]: Committer (1/1) (a364a10472f75b65b54f92b065f3617c_20ba6b65f97481d5570070de90e4e791_0_0) switched from RUNNING to FINISHED.
2024-09-25 22:38:46,014 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager - Clearing resource requirements of job eee7e6b18a8ea0778e05d68efd0520ae
2024-09-25 22:38:46,017 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job insert-into_default_catalog.default_database.sink (eee7e6b18a8ea0778e05d68efd0520ae) switched from state RUNNING to FINISHED.
2024-09-25 22:38:46,018 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:46,023 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad - stream load started for zzz_label_0_test_primary3_fa3dd5cc-c489-48a1-adb4-6eaff9869405 on host xxxx
2024-09-25 22:38:46,029 INFO org.apache.flink.runtime.minicluster.MiniCluster - Shutting down Flink Mini Cluster
2024-09-25 22:38:46,029 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job eee7e6b18a8ea0778e05d68efd0520ae reached terminal state FINISHED.
2024-09-25 22:38:46,030 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopping TaskExecutor akka://flink/user/rpc/taskmanager_0.
2024-09-25 22:38:46,030 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager connection cd7181f70a77f5e55a066c559abd7953.
2024-09-25 22:38:46,031 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Closing TaskExecutor connection 4eb7c128-3895-4260-8d21-ebaf9dd3ac92 because: The TaskExecutor is shutting down.
2024-09-25 22:38:46,031 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job eee7e6b18a8ea0778e05d68efd0520ae.
2024-09-25 22:38:46,032 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job eee7e6b18a8ea0778e05d68efd0520ae has been registered for cleanup in the JobResultStore after reaching a terminal state.
2024-09-25 22:38:46,033 INFO org.apache.flink.runtime.state.TaskExecutorStateChangelogStoragesManager - Shutting down TaskExecutorStateChangelogStoragesManager.
2024-09-25 22:38:46,034 INFO org.apache.flink.runtime.jobmaster.JobMaster - Disconnect TaskExecutor 4eb7c128-3895-4260-8d21-ebaf9dd3ac92 because: The TaskExecutor is shutting down.
2024-09-25 22:38:46,034 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Free slot TaskSlot(index:0, state:ALLOCATED, resource profile: ResourceProfile{taskHeapMemory=1024.000gb (1099511627776 bytes), taskOffHeapMemory=1024.000gb (1099511627776 bytes), managedMemory=128.000mb (134217728 bytes), networkMemory=64.000mb (67108864 bytes)}, allocationId: fd7235e8279573693eed5b6e8f0c866d, jobId: eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:46,035 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shutting down rest endpoint.
2024-09-25 22:38:46,035 INFO org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool - Releasing slot [fd7235e8279573693eed5b6e8f0c866d].
2024-09-25 22:38:46,036 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:46,038 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2024-09-25 22:38:46,039 INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection cd7181f70a77f5e55a066c559abd7953: Stopping JobMaster for job 'insert-into_default_catalog.default_database.sink' (eee7e6b18a8ea0778e05d68efd0520ae).
2024-09-25 22:38:46,041 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Stop job leader service.
2024-09-25 22:38:46,041 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager - Shutting down TaskExecutorLocalStateStoresManager.
2024-09-25 22:38:46,042 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Disconnect job manager a09b7b27564493391dc36010c42d48dd@akka://flink/user/rpc/jobmanager_3 for job eee7e6b18a8ea0778e05d68efd0520ae from the resource manager.
2024-09-25 22:38:46,051 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Removing cache directory C:\Users\zz\AppData\Local\Temp\flink-web-ui
2024-09-25 22:38:46,052 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager removed spill file directory C:\Users\zz\AppData\Local\Temp\flink-io-9bcf4fd8-0a9f-4b22-a7f8-0d6239767bf9
2024-09-25 22:38:46,052 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Shutting down the network environment and its components.
2024-09-25 22:38:46,053 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shut down complete.
2024-09-25 22:38:46,054 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl - FileChannelManager removed spill file directory C:\Users\zz\AppData\Local\Temp\flink-netty-shuffle-55a4a9d0-2286-4196-a134-e25efbac28dd
2024-09-25 22:38:46,054 INFO org.apache.flink.runtime.taskexecutor.KvStateService - Shutting down the kvState service and its components.
2024-09-25 22:38:46,054 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService - Stop job leader service.
2024-09-25 22:38:46,055 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Shut down cluster because application is in CANCELED, diagnostics DispatcherResourceManagerComponent has been closed..
2024-09-25 22:38:46,055 INFO org.apache.flink.runtime.filecache.FileCache - removed file cache directory C:\Users\zz\AppData\Local\Temp\flink-dist-cache-43c9b1e7-dd62-42f5-899e-ef0cd563f112
2024-09-25 22:38:46,055 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent - Closing components.
2024-09-25 22:38:46,056 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopped TaskExecutor akka://flink/user/rpc/taskmanager_0.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess - Stopping SessionDispatcherLeaderProcess.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping dispatcher akka://flink/user/rpc/dispatcher_2.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl - Stopping resource manager service.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping all currently running jobs of dispatcher akka://flink/user/rpc/dispatcher_2.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager - Closing the slot manager.
2024-09-25 22:38:46,057 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager - Suspending the slot manager.
2024-09-25 22:38:46,058 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopped dispatcher akka://flink/user/rpc/dispatcher_2.
2024-09-25 22:38:46,059 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service.
2024-09-25 22:38:46,112 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service.
2024-09-25 22:38:46,112 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service.
2024-09-25 22:38:46,123 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache
2024-09-25 22:38:46,124 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
2024-09-25 22:38:46,125 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:4723
2024-09-25 22:38:46,130 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service.
2024-09-25 22:38:46,453 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad - load Result {
"TxnId": 7144,
"Label": "zzz_label_0_test_primary3_fa3dd5cc-c489-48a1-adb4-6eaff9869405",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 6,
"LoadTimeMs": 99,
"BeginTxnTimeMs": 1,
"StreamLoadPutTimeMs": 7,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 63,
"CommitAndPublishTimeMs": 26
}
2024-09-25 22:38:46,456 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad - load success, cacheBeforeFlushBytes: 6, currentCacheBytes : 0
2024-09-25 22:38:46,456 INFO org.apache.doris.flink.sink.batch.DorisBatchStreamLoad - LoadAsyncExecutor stop
