从2.1.3升级到2.1.5版本运行一段时间后,定时任务调度的sql卡住,重启fe恢复正常

Viewed 114

2.1.5版本运行一段时间后,定时任务调度的sql卡住,卡住的基本都是insert语句,集群资源利用率不高,排除资源不够的问题。fe重启后恢复正常。卡住后fe有大量如下警告:

2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:53816, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:36692, query timeout: 14400000, query id: TUniqueId(hi:2486096548944561888, lo:-7680119310084599339)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:36692, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:58882, query timeout: 14400000, query id: TUniqueId(hi:5329622498865006451, lo:-5940034225027432946)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:58882, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:55620, query timeout: 14400000, query id: TUniqueId(hi:-6235778904596854579, lo:-6762043617299625790)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:55620, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:55366, query timeout: 14400000, query id: TUniqueId(hi:-7489987598881505095, lo:-5904518638302150014)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:55366, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:46806, query timeout: 14400000, query id: TUniqueId(hi:-1681854942988513629, lo:-8995556750334127387)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:46806, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:47576, query timeout: 14400000, query id: TUniqueId(hi:2893173062655954147, lo:-5525350557404474375)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:47576, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:37050, query timeout: 14400000, query id: TUniqueId(hi:6116423421140747349, lo:-7738397350363445376)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:37050, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:51822, query timeout: 14400000, query id: TUniqueId(hi:-8105407478540058126, lo:-8287084406501610976)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:51822, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill insert timeout, remote: 10.223.225.164:52252, query timeout: 14400000, query id: TUniqueId(hi:5394603519271717004, lo:-6645156992536066479)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.164:52252, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill query timeout, remote: 10.223.225.206:51966, query timeout: 900000, query id: TUniqueId(hi:-480889526106370828, lo:-7707350827635970246)
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.206:51966, kill mysql connection: false reason time out
2024-08-30 08:58:25,641 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.checkTimeout():962] kill query timeout, remote: 10.223.225.206:34906, query timeout: 900000, query id: TUniqueId(hi:-1798157553917146538, lo:-4768306447972261454)
2024-08-30 08:58:25,642 WARN (connect-scheduler-check-timer-0|201) [ConnectContext.killByTimeout():913] kill query from 10.223.225.206:34906, kill mysql connection: false reason time out

fe的jstack线程栈已上传到百度网盘
链接: https://pan.baidu.com/s/1g4vQJZqUYVQ6f5hrcQQu0g 提取码: s5yc

1 Answers

1、目前是否还有复现
2、2.1.3有相关lock bug,不过在2.1.5进行了fix