2024-11-21 20:50:59,142 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 99 for job 4885c97fc0f22a214cd1c12e230b2bc0. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2212) [flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2024-11-21 20:50:59,147 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 99 for job 4885c97fc0f22a214cd1c12e230b2bc0. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
at org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2143) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2130) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$700(CheckpointCoordinator.java:101) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2212) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2024-11-21 20:50:59,147 INFO org.apache.flink.runtime.checkpoint.CheckpointRequestDecider [] - checkpoint request time in queue: 1777879
2024-11-21 20:50:59,162 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. The latest checkpoint failed due to Checkpoint expired before completing., view the Checkpoint History tab or the Job Manager log to find out why continuous checkpoints failed.
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:212) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:169) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:122) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2151) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2130) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$700(CheckpointCoordinator.java:101) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2212) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
2024-11-21 20:50:59,169 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - 3 tasks will be restarted to recover from a global failure.
2024-11-21 20:50:59,170 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job MySQL-Doris Sync Database: ai_hub_base_hospital (4885c97fc0f22a214cd1c12e230b2bc0) switched from state RUNNING to RESTARTING.
2024-11-21 20:50:59,178 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL Source -> Process -> (c_auth_application: Writer -> c_auth_application: Committer, c_auth_role: Writer -> c_auth_role: Committer, c_auth_role_authority: Writer -> c_auth_role_authority: Committer, c_auth_role_org: Writer -> c_auth_role_org: Committer, c_auth_system_api: Writer -> c_auth_system_api: Committer, c_auth_user: Writer -> c_auth_user: Committer, c_auth_user_role: Writer -> c_auth_user_role: Committer, c_auth_user_token: Writer -> c_auth_user_token: Committer, c_common_area: Writer -> c_common_area: Committer, c_common_dictionary: Writer -> c_common_dictionary: Committer, c_common_dictionary_item: Writer -> c_common_dictionary_item: Committer, c_common_login_log: Writer -> c_common_login_log: Committer, c_common_opt_log: Writer -> c_common_opt_log: Committer, c_common_parameter: Writer -> c_common_parameter: Committer, c_core_org: Writer -> c_core_org: Committer, c_core_station: Writer -> c_core_station: Committer, c_signature: Writer -> c_signature: Committer, c_tenant_configuration: Writer -> c_tenant_configuration: Committer, flyway_schema_history: Writer -> flyway_schema_history: Committer, f_attachment: Writer -> f_attachment: Committer, f_file: Writer -> f_file: Committer, _analysis_fail_log: Writer -> _analysis_fail_log: Committer, _analysis_request_callback_log: Writer -> _analysis_request_callback_log: Committer, _instance_report: Writer -> _instance_report: Committer, _instance_report_classification: Writer -> _instance_report_classification: Committer, _instance_report_classifications: Writer -> _instance_report_classifications: Committer, _instance_report_detect: Writer -> _instance_report_detect: Committer, _instance_report_detect_coordinate: Writer -> _instance_report_detect_coordinate: Committer, _instance_report_segmentation: Writer -> _instance_report_segmentation: Committer, _instance_report_segmentation_lesion: Writer -> _instance_report_segmentation_lesion: Committer, _instance_report_segmentation_pixel: Writer -> _instance_report_segmentation_pixel: Committer, _report_conclusion: Writer -> _report_conclusion: Committer, _report_generate_lesion_record: Writer -> _report_generate_lesion_record: Committer, _report_instance_record: Writer -> _report_instance_record: Committer, _report_lesion_record: Writer -> _report_lesion_record: Committer, _series_report: Writer -> _series_report: Committer, _series_report_bone_density: Writer -> _series_report_bone_density: Committer, _series_report_bone_density_report: Writer -> _series_report_bone_density_report: Committer, _series_report_check_preview: Writer -> _series_report_check_preview: Committer, _series_report_classification: Writer -> _series_report_classification: Committer, _series_report_coordinate3d: Writer -> _series_report_coordinate3d: Committer, _series_report_detect_coordinate: Writer -> _series_report_detect_coordinate: Committer, _series_report_detect_coordinate_diameter: Writer -> _series_report_detect_coordinate_diameter: Committer, _series_report_detect_coordinate_lung: Writer -> _series_report_detect_coordinate_lung: Committer, _series_report_instance_img: Writer -> _series_report_instance_img: Committer, _series_report_intervertebral: Writer -> _series_report_intervertebral: Committer, _series_report_lesion: Writer -> _series_report_lesion: Committer, _series_report_lesion_arrow: Writer -> _series_report_lesion_arrow: Committer, _series_report_lesion_coordinate3d: Writer -> _series_report_lesion_coordinate3d: Committer, _series_report_lesion_histogram: Writer -> _series_report_lesion_histogram: Committer, _series_report_lesion_img: Writer -> _series_report_lesion_img: Committer, _series_report_lesion_mark_img: Writer -> _series_report_lesion_mark_img: Committer, _series_report_lesion_pixel3d: Writer -> _series_report_lesion_pixel3d: Committer, _series_report_lesion_uid: Writer -> _series_report_lesion_uid: Committer, _series_report_markrib: Writer -> _series_report_markrib: Committer, _series_report_mark_type: Writer -> _series_report_mark_type: Committer, _series_report_morphology_report: Writer -> _series_report_morphology_report: Committer, _series_report_morphology_report_vertebrae: Writer -> _series_report_morphology_report_vertebrae: Committer, _series_report_morphology_report_vertebrae_act: Writer -> _series_report_morphology_report_vertebrae_act: Committer, _series_report_morphology_report_vertebrae_category: Writer -> _series_report_morphology_report_vertebrae_category: Committer, _series_report_morphology_report_vertebrae_mark: Writer -> _series_report_morphology_report_vertebrae_mark: Committer, _series_report_musclefat: Writer -> _series_report_musclefat: Committer, _series_report_pixel3d: Writer -> _series_report_pixel3d: Committer, _series_report_preview: Writer -> _series_report_preview: Committer, _series_report_segmentation_pixel: Writer -> _series_report_segmentation_pixel: Committer, _series_report_segmentation_pixel_measurement: Writer -> _series_report_segmentation_pixel_measurement: Committer, _series_report_segmentation_pixel_reference: Writer -> _series_report_segmentation_pixel_reference: Committer, _series_report_tissue: Writer -> _series_report_tissue: Committer, _study_report: Writer -> _study_report: Committer, _study_report_mask: Writer -> _study_report_mask: Committer, checked_img: Writer -> checked_img: Committer, checked_msg: Writer -> checked_msg: Committer, checked_report: Writer -> checked_report: Committer, disease_category: Writer -> disease_category: Committer, doctor_handle_detail_log: Writer -> doctor_handle_detail_log: Committer, doctor_patient_data_relation: Writer -> doctor_patient_data_relation: Committer, doctor_used_disease_category: Writer -> doctor_used_disease_category: Committer, patient_auto_body_series_description_rule: Writer -> patient_auto_body_series_description_rule: Committer, patient_auto_series_description: Writer -> patient_auto_series_description: Committer, patient_auto_series_rule: Writer -> patient_auto_series_rule: Committer, patient_checked: Writer -> patient_checked: Committer, patient_info: Writer -> patient_info: Committer, patient_instance: Writer -> patient_instance: Committer, patient_series: Writer -> patient_series: Committer, patient_series_auto_diagnosis_body: Writer -> patient_series_auto_diagnosis_body: Committer, patient_series_diagnosis: Writer -> patient_series_diagnosis: Committer, patient_study: Writer -> patient_study: Committer, plugin_usage_record: Writer -> plugin_usage_record: Committer, report_healthy_advise: Writer -> report_healthy_advise: Committer, report_msg: Writer -> report_msg: Committer, screenshot_request_callback_log: Writer -> screenshot_request_callback_log: Committer, series_calibration: Writer -> series_calibration: Committer, series_name_convert_config: Writer -> series_name_convert_config: Committer, series_name_no_convert_body: Writer -> series_name_no_convert_body: Committer, statistics: Writer -> statistics: Committer, upload_file_detail_log: Writer -> upload_file_detail_log: Committer, mail_provider: Writer -> mail_provider: Committer, mail_send_status: Writer -> mail_send_status: Committer, mail_task: Writer -> mail_task: Committer, msgs_center_info: Writer -> msgs_center_info: Committer, msgs_center_info_receive: Writer -> msgs_center_info_receive: Committer, sms_send_status: Writer -> sms_send_status: Committer, sms_task: Writer -> sms_task: Committer, sms_template: Writer -> sms_template: Committer, undo_log: Writer -> undo_log: Committer) (3/3) (1f8e720de8a45ff1815d44bbefddfa3b_cbc357ccb763df2852fee8c4fc7d55f2_2_0) switched from RUNNING to CANCELING.
2024-11-21 20:50:59,183 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL Source -> Process -> (c_auth_application: Writer -> c_auth_application: Committer, c_auth_role: Writer -> c_auth_role: Committer, c_auth_role_authority: Writer -> c_auth_role_authority: Committer, c_auth_role_org: Writer -> c_auth_role_org: Committer, c_auth_system_api: Writer -> c_auth_system_api: Committer, c_auth_user: Writer -> c_auth_user: Committer, c_auth_user_role: Writer -> c_auth_user_role: Committer, c_auth_user_token: Writer -> c_auth_user_token: Committer, c_common_area: Writer -> c_common_area: Committer, c_common_dictionary: Writer -> c_common_dictionary: Committer, c_common_dictionary_item: Writer -> c_common_dictionary_item: Committer, c_common_login_log: Writer -> c_common_login_log: Committer, c_common_opt_log: Writer -> c_common_opt_log: Committer, c_common_parameter: Writer -> c_common_parameter: Committer, c_core_org: Writer -> c_core_org: Committer, c_core_station: Writer -> c_core_station: Committer, c_signature: Writer -> c_signature: Committer, c_tenant_configuration: Writer -> c_tenant_configuration: Committer, flyway_schema_history: Writer -> flyway_schema_history: Committer, f_attachment: Writer -> f_attachment: Committer, f_file: Writer -> f_file: Committer, _analysis_fail_log: Writer -> _analysis_fail_log: Committer, _analysis_request_callback_log: Writer -> _analysis_request_callback_log: Committer, _instance_report: Writer -> _instance_report: Committer, _instance_report_classification: Writer -> _instance_report_classification: Committer, _instance_report_classifications: Writer -> _instance_report_classifications: Committer, _instance_report_detect: Writer -> _instance_report_detect: Committer, _instance_report_detect_coordinate: Writer -> _instance_report_detect_coordinate: Committer, _instance_report_segmentation: Writer -> _instance_report_segmentation: Committer, _instance_report_segmentation_lesion: Writer -> _instance_report_segmentation_lesion: Committer, _instance_report_segmentation_pixel: Writer -> _instance_report_segmentation_pixel: Committer, _report_conclusion: Writer -> _report_conclusion: Committer, _report_generate_lesion_record: Writer -> _report_generate_lesion_record: Committer, _report_instance_record: Writer -> _report_instance_record: Committer, _report_lesion_record: Writer -> _report_lesion_record: Committer, _series_report: Writer -> _series_report: Committer, _series_report_bone_density: Writer -> _series_report_bone_density: Committer, _series_report_bone_density_report: Writer -> _series_report_bone_density_report: Committer, _series_report_check_preview: Writer -> _series_report_check_preview: Committer, _series_report_classification: Writer -> _series_report_classification: Committer, _series_report_coordinate3d: Writer -> _series_report_coordinate3d: Committer, _series_report_detect_coordinate: Writer -> _series_report_detect_coordinate: Committer, _series_report_detect_coordinate_diameter: Writer -> _series_report_detect_coordinate_diameter: Committer, _series_report_detect_coordinate_lung: Writer -> _series_report_detect_coordinate_lung: Committer, _series_report_instance_img: Writer -> _series_report_instance_img: Committer, _series_report_intervertebral: Writer -> _series_report_intervertebral: Committer, _series_report_lesion: Writer -> _series_report_lesion: Committer, _series_report_lesion_arrow: Writer -> _series_report_lesion_arrow: Committer, _series_report_lesion_coordinate3d: Writer -> _series_report_lesion_coordinate3d: Committer, _series_report_lesion_histogram: Writer -> _series_report_lesion_histogram: Committer, _series_report_lesion_img: Writer -> _series_report_lesion_img: Committer, _series_report_lesion_mark_img: Writer -> _series_report_lesion_mark_img: Committer, _series_report_lesion_pixel3d: Writer -> _series_report_lesion_pixel3d: Committer, _series_report_lesion_uid: Writer -> _series_report_lesion_uid: Committer, _series_report_markrib: Writer -> _series_report_markrib: Committer, _series_report_mark_type: Writer -> _series_report_mark_type: Committer, _series_report_morphology_report: Writer -> _series_report_morphology_report: Committer, _series_report_morphology_report_vertebrae: Writer -> _series_report_morphology_report_vertebrae: Committer, _series_report_morphology_report_vertebrae_act: Writer -> _series_report_morphology_report_vertebrae_act: Committer, _series_report_morphology_report_vertebrae_category: Writer -> _series_report_morphology_report_vertebrae_category: Committer, _series_report_morphology_report_vertebrae_mark: Writer -> _series_report_morphology_report_vertebrae_mark: Committer, _series_report_musclefat: Writer -> _series_report_musclefat: Committer, _series_report_pixel3d: Writer -> _series_report_pixel3d: Committer, _series_report_preview: Writer -> _series_report_preview: Committer, _series_report_segmentation_pixel: Writer -> _series_report_segmentation_pixel: Committer, _series_report_segmentation_pixel_measurement: Writer -> _series_report_segmentation_pixel_measurement: Committer, _series_report_segmentation_pixel_reference: Writer -> _series_report_segmentation_pixel_reference: Committer, _series_report_tissue: Writer -> _series_report_tissue: Committer, _study_report: Writer -> _study_report: Committer, _study_report_mask: Writer -> _study_report_mask: Committer, checked_img: Writer -> checked_img: Committer, checked_msg: Writer -> checked_msg: Committer, checked_report: Writer -> checked_report: Committer, disease_category: Writer -> disease_category: Committer, doctor_handle_detail_log: Writer -> doctor_handle_detail_log: Committer, doctor_patient_data_relation: Writer -> doctor_patient_data_relation: Committer, doctor_used_disease_category: Writer -> doctor_used_disease_category: Committer, patient_auto_body_series_description_rule: Writer -> patient_auto_body_series_description_rule: Committer, patient_auto_series_description: Writer -> patient_auto_series_description: Committer, patient_auto_series_rule: Writer -> patient_auto_series_rule: Committer, patient_checked: Writer -> patient_checked: Committer, patient_info: Writer -> patient_info: Committer, patient_instance: Writer -> patient_instance: Committer, patient_series: Writer -> patient_series: Committer, patient_series_auto_diagnosis_body: Writer -> patient_series_auto_diagnosis_body: Committer, patient_series_diagnosis: Writer -> patient_series_diagnosis: Committer, patient_study: Writer -> patient_study: Committer, plugin_usage_record: Writer -> plugin_usage_record: Committer, report_healthy_advise: Writer -> report_healthy_advise: Committer, report_msg: Writer -> report_msg: Committer, screenshot_request_callback_log: Writer -> screenshot_request_callback_log: Committer, series_calibration: Writer -> series_calibration: Committer, series_name_convert_config: Writer -> series_name_convert_config: Committer, series_name_no_convert_body: Writer -> series_name_no_convert_body: Committer, statistics: Writer -> statistics: Committer, upload_file_detail_log: Writer -> upload_file_detail_log: Committer, mail_provider: Writer -> mail_provider: Committer, mail_send_status: Writer -> mail_send_status: Committer, mail_task: Writer -> mail_task: Committer, msgs_center_info: Writer -> msgs_center_info: Committer, msgs_center_info_receive: Writer -> msgs_center_info_receive: Committer, sms_send_status: Writer -> sms_send_status: Committer, sms_task: Writer -> sms_task: Committer, sms_template: Writer -> sms_template: Committer, undo_log: Writer -> undo_log: Committer) (2/3) (1f8e720de8a45ff1815d44bbefddfa3b_cbc357ccb763df2852fee8c4fc7d55f2_1_0) switched from RUNNING to CANCELING.
2024-11-21 20:50:59,184 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL Source -> Process -> (c_auth_application: Writer -> c_auth_application: Committer, c_auth_role: Writer -> c_auth_role: Committer, c_auth_role_authority: Writer -> c_auth_role_authority: Committer, c_auth_role_org: Writer -> c_auth_role_org: Committer, c_auth_system_api: Writer -> c_auth_system_api: Committer, c_auth_user: Writer -> c_auth_user: Committer, c_auth_user_role: Writer -> c_auth_user_role: Committer, c_auth_user_token: Writer -> c_auth_user_token: Committer, c_common_area: Writer -> c_common_area: Committer, c_common_dictionary: Writer -> c_common_dictionary: Committer, c_common_dictionary_item: Writer -> c_common_dictionary_item: Committer, c_common_login_log: Writer -> c_common_login_log: Committer, c_common_opt_log: Writer -> c_common_opt_log: Committer, c_common_parameter: Writer -> c_common_parameter: Committer, c_core_org: Writer -> c_core_org: Committer, c_core_station: Writer -> c_core_station: Committer, c_signature: Writer -> c_signature: Committer, c_tenant_configuration: Writer -> c_tenant_configuration: Committer, flyway_schema_history: Writer -> flyway_schema_history: Committer, f_attachment: Writer -> f_attachment: Committer, f_file: Writer -> f_file: Committer, _analysis_fail_log: Writer -> _analysis_fail_log: Committer, _analysis_request_callback_log: Writer -> _analysis_request_callback_log: Committer, _instance_report: Writer -> _instance_report: Committer, _instance_report_classification: Writer -> _instance_report_classification: Committer, _instance_report_classifications: Writer -> _instance_report_classifications: Committer, _instance_report_detect: Writer -> _instance_report_detect: Committer, _instance_report_detect_coordinate: Writer -> _instance_report_detect_coordinate: Committer, _instance_report_segmentation: Writer -> _instance_report_segmentation: Committer, _instance_report_segmentation_lesion: Writer -> _instance_report_segmentation_lesion: Committer, _instance_report_segmentation_pixel: Writer -> _instance_report_segmentation_pixel: Committer, _report_conclusion: Writer -> _report_conclusion: Committer, _report_generate_lesion_record: Writer -> _report_generate_lesion_record: Committer, _report_instance_record: Writer -> _report_instance_record: Committer, _report_lesion_record: Writer -> _report_lesion_record: Committer, _series_report: Writer -> _series_report: Committer, _series_report_bone_density: Writer -> _series_report_bone_density: Committer, _series_report_bone_density_report: Writer -> _series_report_bone_density_report: Committer, _series_report_check_preview: Writer -> _series_report_check_preview: Committer, _series_report_classification: Writer -> _series_report_classification: Committer, _series_report_coordinate3d: Writer -> _series_report_coordinate3d: Committer, _series_report_detect_coordinate: Writer -> _series_report_detect_coordinate: Committer, _series_report_detect_coordinate_diameter: Writer -> _series_report_detect_coordinate_diameter: Committer, _series_report_detect_coordinate_lung: Writer -> _series_report_detect_coordinate_lung: Committer, _series_report_instance_img: Writer -> _series_report_instance_img: Committer, _series_report_intervertebral: Writer -> _series_report_intervertebral: Committer, _series_report_lesion: Writer -> _series_report_lesion: Committer, _series_report_lesion_arrow: Writer -> _series_report_lesion_arrow: Committer, _series_report_lesion_coordinate3d: Writer -> _series_report_lesion_coordinate3d: Committer, _series_report_lesion_histogram: Writer -> _series_report_lesion_histogram: Committer, _series_report_lesion_img: Writer -> _series_report_lesion_img: Committer, _series_report_lesion_mark_img: Writer -> _series_report_lesion_mark_img: Committer, _series_report_lesion_pixel3d: Writer -> _series_report_lesion_pixel3d: Committer, _series_report_lesion_uid: Writer -> _series_report_lesion_uid: Committer, _series_report_markrib: Writer -> _series_report_markrib: Committer, _series_report_mark_type: Writer -> _series_report_mark_type: Committer, _series_report_morphology_report: Writer -> _series_report_morphology_report: Committer, _series_report_morphology_report_vertebrae: Writer -> _series_report_morphology_report_vertebrae: Committer, _series_report_morphology_report_vertebrae_act: Writer -> _series_report_morphology_report_vertebrae_act: Committer, _series_report_morphology_report_vertebrae_category: Writer -> _series_report_morphology_report_vertebrae_category: Committer, _series_report_morphology_report_vertebrae_mark: Writer -> _series_report_morphology_report_vertebrae_mark: Committer, _series_report_musclefat: Writer -> _series_report_musclefat: Committer, _series_report_pixel3d: Writer -> _series_report_pixel3d: Committer, _series_report_preview: Writer -> _series_report_preview: Committer, _series_report_segmentation_pixel: Writer -> _series_report_segmentation_pixel: Committer, _series_report_segmentation_pixel_measurement: Writer -> _series_report_segmentation_pixel_measurement: Committer, _series_report_segmentation_pixel_reference: Writer -> _series_report_segmentation_pixel_reference: Committer, _series_report_tissue: Writer -> _series_report_tissue: Committer, _study_report: Writer -> _study_report: Committer, _study_report_mask: Writer -> _study_report_mask: Committer, checked_img: Writer -> checked_img: Committer, checked_msg: Writer -> checked_msg: Committer, checked_report: Writer -> checked_report: Committer, disease_category: Writer -> disease_category: Committer, doctor_handle_detail_log: Writer -> doctor_handle_detail_log: Committer, doctor_patient_data_relation: Writer -> doctor_patient_data_relation: Committer, doctor_used_disease_category: Writer -> doctor_used_disease_category: Committer, patient_auto_body_series_description_rule: Writer -> patient_auto_body_series_description_rule: Committer, patient_auto_series_description: Writer -> patient_auto_series_description: Committer, patient_auto_series_rule: Writer -> patient_auto_series_rule: Committer, patient_checked: Writer -> patient_checked: Committer, patient_info: Writer -> patient_info: Committer, patient_instance: Writer -> patient_instance: Committer, patient_series: Writer -> patient_series: Committer, patient_series_auto_diagnosis_body: Writer -> patient_series_auto_diagnosis_body: Committer, patient_series_diagnosis: Writer -> patient_series_diagnosis: Committer, patient_study: Writer -> patient_study: Committer, plugin_usage_record: Writer -> plugin_usage_record: Committer, report_healthy_advise: Writer -> report_healthy_advise: Committer, report_msg: Writer -> report_msg: Committer, screenshot_request_callback_log: Writer -> screenshot_request_callback_log: Committer, series_calibration: Writer -> series_calibration: Committer, series_name_convert_config: Writer -> series_name_convert_config: Committer, series_name_no_convert_body: Writer -> series_name_no_convert_body: Committer, statistics: Writer -> statistics: Committer, upload_file_detail_log: Writer -> upload_file_detail_log: Committer, mail_provider: Writer -> mail_provider: Committer, mail_send_status: Writer -> mail_send_status: Committer, mail_task: Writer -> mail_task: Committer, msgs_center_info: Writer -> msgs_center_info: Committer, msgs_center_info_receive: Writer -> msgs_center_info_receive: Committer, sms_send_status: Writer -> sms_send_status: Committer, sms_task: Writer -> sms_task: Committer, sms_template: Writer -> sms_template: Committer, undo_log: Writer -> undo_log: Committer) (1/3) (1f8e720de8a45ff1815d44bbefddfa3b_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to CANCELING.
2024-11-21 20:50:59,195 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. The latest checkpoint failed due to Checkpoint expired before completing., view the Checkpoint History tab or the Job Manager log to find out why continuous checkpoints failed.
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:212) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:169) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:122) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:997) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:967) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:695) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[?:1.8.0_181]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
2024-11-21 20:50:59,195 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - 3 tasks will be restarted to recover from a global failure.
2024-11-21 20:52:01,728 ERROR org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator [] - Failed to reset the coordinator to checkpoint and start.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@26aad3d7 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5fdae327[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) ~[?:1.8.0_181]
at org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext.runInCoordinatorThread(SourceCoordinatorContext.java:342) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.runInEventLoop(SourceCoordinator.java:466) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:233) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.resetAndStart(RecreateOnResetOperatorCoordinator.java:416) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.lambda$resetToCheckpoint$7(RecreateOnResetOperatorCoordinator.java:156) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_181]
at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1093) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.util.concurrent.Executors$DirectExecutor.execute(Executors.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:457) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2024-11-21 20:52:01,729 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: MySQL Source closed.
2024-11-21 20:52:01,732 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: MySQL Source -> Process -> (c_auth_application: Writer -> c_auth_application: Committer, c_auth_role: Writer -> c_auth_role: Committer, c_auth_role_authority: Writer -> c_auth_role_authority: Committer, c_auth_role_org: Writer -> c_auth_role_org: Committer, c_auth_system_api: Writer -> c_auth_system_api: Committer, c_auth_user: Writer -> c_auth_user: Committer, c_auth_user_role: Writer -> c_auth_user_role: Committer, c_auth_user_token: Writer -> c_auth_user_token: Committer, c_common_area: Writer -> c_common_area: Committer, c_common_dictionary: Writer -> c_common_dictionary: Committer, c_common_dictionary_item: Writer -> c_common_dictionary_item: Committer, c_common_login_log: Writer -> c_common_login_log: Committer, c_common_opt_log: Writer -> c_common_opt_log: Committer, c_common_parameter: Writer -> c_common_parameter: Committer, c_core_org: Writer -> c_core_org: Committer, c_core_station: Writer -> c_core_station: Committer, c_signature: Writer -> c_signature: Committer, c_tenant_configuration: Writer -> c_tenant_configuration: Committer, flyway_schema_history: Writer -> flyway_schema_history: Committer, f_attachment: Writer -> f_attachment: Committer, f_file: Writer -> f_file: Committer, _analysis_fail_log: Writer -> _analysis_fail_log: Committer, _analysis_request_callback_log: Writer -> _analysis_request_callback_log: Committer, _instance_report: Writer -> _instance_report: Committer, _instance_report_classification: Writer -> _instance_report_classification: Committer, _instance_report_classifications: Writer -> _instance_report_classifications: Committer, _instance_report_detect: Writer -> _instance_report_detect: Committer, _instance_report_detect_coordinate: Writer -> _instance_report_detect_coordinate: Committer, _instance_report_segmentation: Writer -> _instance_report_segmentation: Committer, _instance_report_segmentation_lesion: Writer -> _instance_report_segmentation_lesion: Committer, _instance_report_segmentation_pixel: Writer -> _instance_report_segmentation_pixel: Committer, _report_conclusion: Writer -> _report_conclusion: Committer, _report_generate_lesion_record: Writer -> _report_generate_lesion_record: Committer, _report_instance_record: Writer -> _report_instance_record: Committer, _report_lesion_record: Writer -> _report_lesion_record: Committer, _series_report: Writer -> _series_report: Committer, _series_report_bone_density: Writer -> _series_report_bone_density: Committer, _series_report_bone_density_report: Writer -> _series_report_bone_density_report: Committer, _series_report_check_preview: Writer -> _series_report_check_preview: Committer, _series_report_classification: Writer -> _series_report_classification: Committer, _series_report_coordinate3d: Writer -> _series_report_coordinate3d: Committer, _series_report_detect_coordinate: Writer -> _series_report_detect_coordinate: Committer, _series_report_detect_coordinate_diameter: Writer -> _series_report_detect_coordinate_diameter: Committer, _series_report_detect_coordinate_lung: Writer -> _series_report_detect_coordinate_lung: Committer, _series_report_instance_img: Writer -> _series_report_instance_img: Committer, _series_report_intervertebral: Writer -> _series_report_intervertebral: Committer, _series_report_lesion: Writer -> _series_report_lesion: Committer, _series_report_lesion_arrow: Writer -> _series_report_lesion_arrow: Committer, _series_report_lesion_coordinate3d: Writer -> _series_report_lesion_coordinate3d: Committer, _series_report_lesion_histogram: Writer -> _series_report_lesion_histogram: Committer, _series_report_lesion_img: Writer -> _series_report_lesion_img: Committer, _series_report_lesion_mark_img: Writer -> _series_report_lesion_mark_img: Committer, _series_report_lesion_pixel3d: Writer -> _series_report_lesion_pixel3d: Committer, _series_report_lesion_uid: Writer -> _series_report_lesion_uid: Committer, _series_report_markrib: Writer -> _series_report_markrib: Committer, _series_report_mark_type: Writer -> _series_report_mark_type: Committer, _series_report_morphology_report: Writer -> _series_report_morphology_report: Committer, _series_report_morphology_report_vertebrae: Writer -> _series_report_morphology_report_vertebrae: Committer, _series_report_morphology_report_vertebrae_act: Writer -> _series_report_morphology_report_vertebrae_act: Committer, _series_report_morphology_report_vertebrae_category: Writer -> _series_report_morphology_report_vertebrae_category: Committer, _series_report_morphology_report_vertebrae_mark: Writer -> _series_report_morphology_report_vertebrae_mark: Committer, _series_report_musclefat: Writer -> _series_report_musclefat: Committer, _series_report_pixel3d: Writer -> _series_report_pixel3d: Committer, _series_report_preview: Writer -> _series_report_preview: Committer, _series_report_segmentation_pixel: Writer -> _series_report_segmentation_pixel: Committer, _series_report_segmentation_pixel_measurement: Writer -> _series_report_segmentation_pixel_measurement: Committer, _series_report_segmentation_pixel_reference: Writer -> _series_report_segmentation_pixel_reference: Committer, _series_report_tissue: Writer -> _series_report_tissue: Committer, _study_report: Writer -> _study_report: Committer, _study_report_mask: Writer -> _study_report_mask: Committer, checked_img: Writer -> checked_img: Committer, checked_msg: Writer -> checked_msg: Committer, checked_report: Writer -> checked_report: Committer, disease_category: Writer -> disease_category: Committer, doctor_handle_detail_log: Writer -> doctor_handle_detail_log: Committer, doctor_patient_data_relation: Writer -> doctor_patient_data_relation: Committer, doctor_used_disease_category: Writer -> doctor_used_disease_category: Committer, patient_auto_body_series_description_rule: Writer -> patient_auto_body_series_description_rule: Committer, patient_auto_series_description: Writer -> patient_auto_series_description: Committer, patient_auto_series_rule: Writer -> patient_auto_series_rule: Committer, patient_checked: Writer -> patient_checked: Committer, patient_info: Writer -> patient_info: Committer, patient_instance: Writer -> patient_instance: Committer, patient_series: Writer -> patient_series: Committer, patient_series_auto_diagnosis_body: Writer -> patient_series_auto_diagnosis_body: Committer, patient_series_diagnosis: Writer -> patient_series_diagnosis: Committer, patient_study: Writer -> patient_study: Committer, plugin_usage_record: Writer -> plugin_usage_record: Committer, report_healthy_advise: Writer -> report_healthy_advise: Committer, report_msg: Writer -> report_msg: Committer, screenshot_request_callback_log: Writer -> screenshot_request_callback_log: Committer, series_calibration: Writer -> series_calibration: Committer, series_name_convert_config: Writer -> series_name_convert_config: Committer, series_name_no_convert_body: Writer -> series_name_no_convert_body: Committer, statistics: Writer -> statistics: Committer, upload_file_detail_log: Writer -> upload_file_detail_log: Committer, mail_provider: Writer -> mail_provider: Committer, mail_send_status: Writer -> mail_send_status: Committer, mail_task: Writer -> mail_task: Committer, msgs_center_info: Writer -> msgs_center_info: Committer, msgs_center_info_receive: Writer -> msgs_center_info_receive: Committer, sms_send_status: Writer -> sms_send_status: Committer, sms_task: Writer -> sms_task: Committer, sms_template: Writer -> sms_template: Committer, undo_log: Writer -> undo_log: Committer)' (operator cbc357ccb763df2852fee8c4fc7d55f2).
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:624) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.cleanAndFailJob(RecreateOnResetOperatorCoordinator.java:428) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.resetAndStart(RecreateOnResetOperatorCoordinator.java:420) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.lambda$resetToCheckpoint$7(RecreateOnResetOperatorCoordinator.java:156) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_181]
at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1093) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.util.concurrent.Executors$DirectExecutor.execute(Executors.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:457) ~[flink-dist-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@26aad3d7 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5fdae327[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) ~[?:1.8.0_181]
at org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext.runInCoordinatorThread(SourceCoordinatorContext.java:342) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.runInEventLoop(SourceCoordinator.java:466) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:233) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.resetAndStart(RecreateOnResetOperatorCoordinator.java:416) ~[flink-dist-1.18.0.jar:1.18.0]
