机器配置
CPU:是飞腾5000
操作系统: 麒麟v10 sp3
k8s版本:1.26.16
刀片机, 分配的空间是KVM虚拟化出来的
24核48G
测试路线
docker compose部署、k8s部署、裸机部署
be部分配置
JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx6g -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR/heapdump.hprof -DlogPath=$LOG_DIR/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
priority_networks = 172.20.80.0/24
string_type_length_soft_limit_bytes= 2147483643
doris_scanner_thread_pool_thread_num = 4
doris_scanner_thread_pool_queue_size = 8
max_scan_key_num = 64
pipeline_exec_thread_pool_thread_num = 4
enable_fqdn_mode = true
heartbeat_service_password = Deta2025!
fragment_pool_thread_num_max = 2048
fragment_pool_queue_size = 4096
brpc_num_threads = 256
fe 部分配置
JAVA_OPTS="-Dfile.encoding=UTF-8 -Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx6g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$LOG_DIR/fe.gc.log.$CUR_DATE -Dlog4j2.formatMsgNoLookups=true"
qe_query_timeout_second = 300
qe_slow_log_ms = 5000
priority_networks = 172.20.80.0/24
string_type_hard_limit = 2147483643
table_name_length_limit = 256
metadata_checkpoint_interval_seconds = 3600 # 减少元数据检查点频率
enable_fqdn_mode = true
remote_fragment_exec_timeout_ms = 30000
fe.gc.log 崩溃前的几条
2025-03-24T19:03:08.106+0800: 9450.121: [GC pause (G1 Evacuation Pause) (young)GC Start: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
GC End: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
, 0.0260024 secs]
[Parallel Time: 20.6 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 9450121.9, Avg: 9450122.0, Max: 9450122.1, Diff: 0.2]
[Ext Root Scanning (ms): Min: 5.8, Avg: 6.1, Max: 6.4, Diff: 0.6, Sum: 24.2]
[Update RS (ms): Min: 9.6, Avg: 9.6, Max: 9.7, Diff: 0.1, Sum: 38.4]
[Processed Buffers: Min: 30, Avg: 57.0, Max: 116, Diff: 86, Sum: 228]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 3.9, Avg: 4.2, Max: 4.4, Diff: 0.5, Sum: 16.9]
[Termination (ms): Min: 0.2, Avg: 0.3, Max: 0.4, Diff: 0.2, Sum: 1.3]
[Termination Attempts: Min: 1, Avg: 2.5, Max: 5, Diff: 4, Sum: 10]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 20.2, Avg: 20.3, Max: 20.4, Diff: 0.1, Sum: 81.1]
[GC Worker End (ms): Min: 9450142.1, Avg: 9450142.3, Max: 9450142.4, Diff: 0.2]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.6 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 2.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.1 ms]
[Eden: 86016.0K(86016.0K)->0.0B(81920.0K) Survivors: 4096.0K->6144.0K Heap: 269.5M(384.0M)->186.8M(384.0M)]
[Times: user=0.09 sys=0.00, real=0.02 secs]
2025-03-24T19:03:22.776+0800: 9464.791: [GC pause (G1 Evacuation Pause) (young) (initial-mark)GC Start: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
GC End: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
, 0.0239032 secs]
[Parallel Time: 18.4 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 9464793.1, Avg: 9464793.2, Max: 9464793.4, Diff: 0.3]
[Ext Root Scanning (ms): Min: 8.0, Avg: 8.5, Max: 9.1, Diff: 1.1, Sum: 34.2]
[Update RS (ms): Min: 3.9, Avg: 4.1, Max: 4.4, Diff: 0.5, Sum: 16.4]
[Processed Buffers: Min: 40, Avg: 56.2, Max: 86, Diff: 46, Sum: 225]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 3.5, Avg: 3.9, Max: 4.3, Diff: 0.8, Sum: 15.4]
[Termination (ms): Min: 0.1, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.7]
[Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 16.6, Avg: 16.7, Max: 16.8, Diff: 0.2, Sum: 66.9]
[GC Worker End (ms): Min: 9464809.9, Avg: 9464810.0, Max: 9464810.0, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.4 ms]
[Other: 5.0 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 2.8 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.4 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.1 ms]
[Eden: 81920.0K(81920.0K)->0.0B(81920.0K) Survivors: 6144.0K->6144.0K Heap: 266.8M(384.0M)->186.1M(384.0M)]
[Times: user=0.08 sys=0.00, real=0.02 secs]
2025-03-24T19:03:22.800+0800: 9464.816: [GC concurrent-root-region-scan-start]
2025-03-24T19:03:22.811+0800: 9464.826: [GC concurrent-root-region-scan-end, 0.0108680 secs]
2025-03-24T19:03:22.811+0800: 9464.826: [GC concurrent-mark-start]
2025-03-24T19:03:23.038+0800: 9465.054: [GC concurrent-mark-end, 0.2270705 secs]
2025-03-24T19:03:23.040+0800: 9465.055: [GC remark 2025-03-24T19:03:23.040+0800: 9465.055: [Finalize Marking, 0.0014099 secs] 2025-03-24T19:03:23.041+0800: 9465.057: [GC ref-proc, 0.0011486 secs] 2025-03-24T19:03:23.042+0800: 9465.058: [Unloading, 0.0681576 secs], 0.0719300 secs]
[Times: user=0.27 sys=0.00, real=0.07 secs]
2025-03-24T19:03:23.113+0800: 9465.129: [GC cleanup 187M->187M(384M), 0.0035347 secs]
[Times: user=0.01 sys=0.00, real=0.00 secs]
2025-03-24T19:03:35.503+0800: 9477.518: [GC pause (G1 Evacuation Pause) (young)GC Start: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
GC End: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
, 0.0203222 secs]
[Parallel Time: 15.2 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 9477519.1, Avg: 9477519.3, Max: 9477519.4, Diff: 0.3]
[Ext Root Scanning (ms): Min: 6.3, Avg: 6.5, Max: 6.9, Diff: 0.5, Sum: 26.2]
[Update RS (ms): Min: 4.0, Avg: 4.1, Max: 4.3, Diff: 0.2, Sum: 16.5]
[Processed Buffers: Min: 27, Avg: 55.5, Max: 79, Diff: 52, Sum: 222]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 3.6, Avg: 3.9, Max: 4.2, Diff: 0.5, Sum: 15.7]
[Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.2, Diff: 0.2, Sum: 0.6]
[Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 14.5, Avg: 14.8, Max: 15.0, Diff: 0.4, Sum: 59.2]
[GC Worker End (ms): Min: 9477533.9, Avg: 9477534.1, Max: 9477534.1, Diff: 0.2]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 4.5 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 2.4 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.4 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.1 ms]
[Eden: 81920.0K(81920.0K)->0.0B(12288.0K) Survivors: 6144.0K->6144.0K Heap: 266.1M(384.0M)->187.0M(384.0M)]
[Times: user=0.06 sys=0.01, real=0.03 secs]
2025-03-24T19:03:37.792+0800: 9479.807: [GC pause (G1 Evacuation Pause) (mixed)GC Start: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
GC End: NUMA region verification (id: matched/mismatched/total): 0: 0/0/192
, 0.0305925 secs]
[Parallel Time: 27.8 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 9479807.9, Avg: 9479811.6, Max: 9479818.8, Diff: 10.8]
[Ext Root Scanning (ms): Min: 0.1, Avg: 4.6, Max: 9.3, Diff: 9.2, Sum: 18.6]
[Update RS (ms): Min: 4.2, Avg: 6.9, Max: 9.8, Diff: 5.6, Sum: 27.6]
[Processed Buffers: Min: 1, Avg: 34.2, Max: 95, Diff: 94, Sum: 137]
[Scan RS (ms): Min: 0.0, Avg: 1.4, Max: 2.8, Diff: 2.8, Sum: 5.4]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.2]
[Object Copy (ms): Min: 7.9, Avg: 10.4, Max: 16.6, Diff: 8.7, Sum: 41.7]
[Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 0.8]
[Termination Attempts: Min: 1, Avg: 5.5, Max: 13, Diff: 12, Sum: 22]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 16.5, Avg: 23.6, Max: 27.2, Diff: 10.7, Sum: 94.3]
[GC Worker End (ms): Min: 9479835.0, Avg: 9479835.2, Max: 9479835.3, Diff: 0.2]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.3 ms]
[Other: 2.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.8 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.1 ms]
[Eden: 12288.0K(12288.0K)->0.0B(83968.0K) Survivors: 6144.0K->4096.0K Heap: 199.0M(384.0M)->186.4M(384.0M)]
[Times: user=0.11 sys=0.00, real=0.03 secs]
fe.log 崩溃前日志
RuntimeLogger 2025-03-24 11:03:34,539 INFO (report-thread|204) [ReportHandler.storagePolicyReport():364] backend[10004] reports policies [], report resources: []
RuntimeLogger 2025-03-24 11:03:34,539 INFO (report-thread|204) [ReportHandler.tabletReport():473] backend[10004] reports 1078 tablet(s). report version: 17428048011056
RuntimeLogger 2025-03-24 11:03:34,543 INFO (report-thread|204) [TabletInvertedIndex.tabletReport():377] finished to do tablet diff with backend[10004]. fe tablet num: 1078, backend tablet num: 1078. sync: 0. metaDel: 0. foundInMeta: 1078. migration: 0. backend partition num: 138, backend need update: 0. found invalid transactions 0. found republish transactions 0. tabletToUpdate: 0. need recovery: 0. cost: 3 ms
RuntimeLogger 2025-03-24 11:03:34,544 INFO (report-thread|204) [ReportHandler.tabletReport():579] finished to handle tablet report from backend[10004] cost: 4 ms
RuntimeLogger 2025-03-24 11:03:35,189 INFO (mysql-nio-pool-42|6362) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5724 with label label_2c080caf37de44e3_85eee66ee2c74722 from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:35,189 INFO (mysql-nio-pool-42|6362) [OlapInsertExecutor.beforeExec():197] start insert [label_2c080caf37de44e3_85eee66ee2c74722] with query id 2c080caf37de44e3-85eee66ee2c74722 and txn id 5724
RuntimeLogger 2025-03-24 11:03:35,190 INFO (mysql-nio-pool-42|6362) [Coordinator.execInternal():776] dispatch load job: 2c080caf37de44e3-85eee66ee2c74722 to [TNetworkAddress(hostname:172.20.80.5, port:9060)]
RuntimeLogger 2025-03-24 11:03:35,190 INFO (mysql-nio-pool-42|6362) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10004 is 1
RuntimeLogger 2025-03-24 11:03:35,250 INFO (mysql-nio-pool-42|6362) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5724, label: label_2c080caf37de44e3_85eee66ee2c74722, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814215188, commit time: 1742814215245, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:35,254 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5724, db: 10162
RuntimeLogger 2025-03-24 11:03:35,268 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3713, visibleVersionTime: 1742814215265
RuntimeLogger 2025-03-24 11:03:35,273 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33530, eventTime=1742814215273, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:35,273 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5724, label: label_2c080caf37de44e3_85eee66ee2c74722, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814215188, commit time: 1742814215245, finish time: 1742814215265, reason: successfully, publish times 1, publish result QUORUM_SUCC
RuntimeLogger 2025-03-24 11:03:35,299 INFO (mysql-nio-pool-42|6362) [Coordinator.execInternal():756] dispatch result sink of query 4ed91d6c734440a4-b4c500e6e9d02717 to TNetworkAddress(hostname:172.20.80.6, port:9060)
RuntimeLogger 2025-03-24 11:03:35,313 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5725 with label label_51fc822fa1374cad_8d070e80ed8a54f8 from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:35,314 INFO (mysql-nio-pool-44|6692) [OlapInsertExecutor.beforeExec():197] start insert [label_51fc822fa1374cad_8d070e80ed8a54f8] with query id 51fc822fa1374cad-8d070e80ed8a54f8 and txn id 5725
RuntimeLogger 2025-03-24 11:03:35,314 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():776] dispatch load job: 51fc822fa1374cad-8d070e80ed8a54f8 to [TNetworkAddress(hostname:172.20.80.5, port:9060)]
RuntimeLogger 2025-03-24 11:03:35,315 INFO (mysql-nio-pool-44|6692) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10004 is 1
RuntimeLogger 2025-03-24 11:03:35,320 INFO (mysql-nio-pool-42|6362) [StmtExecutor.handleQueryStmt():1827] Query 4ed91d6c734440a4-b4c500e6e9d02717 finished
RuntimeLogger 2025-03-24 11:03:35,377 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5725, label: label_51fc822fa1374cad_8d070e80ed8a54f8, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814215312, commit time: 1742814215373, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:35,377 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5725, db: 10162
RuntimeLogger 2025-03-24 11:03:35,392 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3714, visibleVersionTime: 1742814215388
RuntimeLogger 2025-03-24 11:03:35,395 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33532, eventTime=1742814215395, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:35,396 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5725, label: label_51fc822fa1374cad_8d070e80ed8a54f8, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814215312, commit time: 1742814215373, finish time: 1742814215388, reason: successfully, publish times 1, publish result QUORUM_SUCC
RuntimeLogger 2025-03-24 11:03:35,473 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():756] dispatch result sink of query ead4d5064eb4455-88e0d473fc662532 to TNetworkAddress(hostname:172.20.80.7, port:9060)
RuntimeLogger 2025-03-24 11:03:35,488 INFO (mysql-nio-pool-44|6692) [StmtExecutor.handleQueryStmt():1827] Query ead4d5064eb4455-88e0d473fc662532 finished
RuntimeLogger 2025-03-24 11:03:35,525 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():756] dispatch result sink of query b3f6e298bd224228-a89566627c54434b to TNetworkAddress(hostname:172.20.80.6, port:9060)
RuntimeLogger 2025-03-24 11:03:35,550 INFO (mysql-nio-pool-44|6692) [StmtExecutor.handleQueryStmt():1827] Query b3f6e298bd224228-a89566627c54434b finished
RuntimeLogger 2025-03-24 11:03:36,037 INFO (binlog-gcer|48) [BinlogManager.gc():401] begin gc binlog
RuntimeLogger 2025-03-24 11:03:36,037 INFO (binlog-gcer|48) [BinlogManager.gc():412] gc binlog, dbBinlogMap is null
RuntimeLogger 2025-03-24 11:03:36,037 INFO (binlog-gcer|48) [BinlogGcer.runAfterCatalogReady():63] no gc binlog
RuntimeLogger 2025-03-24 11:03:37,572 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():756] dispatch result sink of query cc38c75223a14005-92e9771cd9ed280e to TNetworkAddress(hostname:172.20.80.7, port:9060)
RuntimeLogger 2025-03-24 11:03:37,573 INFO (mysql-nio-pool-42|6362) [Coordinator.execInternal():756] dispatch result sink of query 1c3f1f0999fc4ccc-aa1bade62fe22726 to TNetworkAddress(hostname:172.20.80.5, port:9060)
RuntimeLogger 2025-03-24 11:03:37,597 INFO (mysql-nio-pool-44|6692) [StmtExecutor.handleQueryStmt():1827] Query cc38c75223a14005-92e9771cd9ed280e finished
RuntimeLogger 2025-03-24 11:03:37,597 INFO (mysql-nio-pool-42|6362) [StmtExecutor.handleQueryStmt():1827] Query 1c3f1f0999fc4ccc-aa1bade62fe22726 finished
RuntimeLogger 2025-03-24 11:03:37,739 INFO (mysql-nio-pool-42|6362) [Coordinator.execInternal():756] dispatch result sink of query 158bcd7d1af84f5d-98dbd9769e13f98d to TNetworkAddress(hostname:172.20.80.7, port:9060)
RuntimeLogger 2025-03-24 11:03:37,751 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():756] dispatch result sink of query 7788aa7db7d84dad-bad990781ca0a9eb to TNetworkAddress(hostname:172.20.80.6, port:9060)
RuntimeLogger 2025-03-24 11:03:37,754 INFO (mysql-nio-pool-42|6362) [StmtExecutor.handleQueryStmt():1827] Query 158bcd7d1af84f5d-98dbd9769e13f98d finished
RuntimeLogger 2025-03-24 11:03:37,769 INFO (mysql-nio-pool-42|6362) [Coordinator.execInternal():756] dispatch result sink of query 20a7dfad245c4a2d-8e53c1232f492901 to TNetworkAddress(hostname:172.20.80.5, port:9060)
RuntimeLogger 2025-03-24 11:03:37,773 INFO (mysql-nio-pool-44|6692) [StmtExecutor.handleQueryStmt():1827] Query 7788aa7db7d84dad-bad990781ca0a9eb finished
RuntimeLogger 2025-03-24 11:03:37,783 INFO (mysql-nio-pool-42|6362) [StmtExecutor.handleQueryStmt():1827] Query 20a7dfad245c4a2d-8e53c1232f492901 finished
RuntimeLogger 2025-03-24 11:03:37,788 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():756] dispatch result sink of query e432bf185af343b0-9f9e24b37c8fe170 to TNetworkAddress(hostname:172.20.80.7, port:9060)
RuntimeLogger 2025-03-24 11:03:37,849 INFO (mysql-nio-pool-44|6692) [StmtExecutor.handleQueryStmt():1827] Query e432bf185af343b0-9f9e24b37c8fe170 finished
RuntimeLogger 2025-03-24 11:03:37,989 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5726 with label label_5094b552cc804742_a6fc243eec955356 from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:37,990 INFO (mysql-nio-pool-44|6692) [OlapInsertExecutor.beforeExec():197] start insert [label_5094b552cc804742_a6fc243eec955356] with query id 5094b552cc804742-a6fc243eec955356 and txn id 5726
RuntimeLogger 2025-03-24 11:03:37,990 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():776] dispatch load job: 5094b552cc804742-a6fc243eec955356 to [TNetworkAddress(hostname:172.20.80.6, port:9060)]
RuntimeLogger 2025-03-24 11:03:37,991 INFO (mysql-nio-pool-44|6692) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10006 is 1
RuntimeLogger 2025-03-24 11:03:38,047 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5726, label: label_5094b552cc804742_a6fc243eec955356, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814217988, commit time: 1742814218044, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:38,054 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5726, db: 10162
RuntimeLogger 2025-03-24 11:03:38,067 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3715, visibleVersionTime: 1742814218064
RuntimeLogger 2025-03-24 11:03:38,071 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33534, eventTime=1742814218071, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:38,071 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5726, label: label_5094b552cc804742_a6fc243eec955356, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814217988, commit time: 1742814218044, finish time: 1742814218064, reason: successfully, publish times 1, publish result QUORUM_SUCC
RuntimeLogger 2025-03-24 11:03:38,178 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5727 with label label_561988f831964eb0_a11e1c4d694f2861 from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:38,179 INFO (mysql-nio-pool-44|6692) [OlapInsertExecutor.beforeExec():197] start insert [label_561988f831964eb0_a11e1c4d694f2861] with query id 561988f831964eb0-a11e1c4d694f2861 and txn id 5727
RuntimeLogger 2025-03-24 11:03:38,179 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():776] dispatch load job: 561988f831964eb0-a11e1c4d694f2861 to [TNetworkAddress(hostname:172.20.80.5, port:9060)]
RuntimeLogger 2025-03-24 11:03:38,180 INFO (mysql-nio-pool-44|6692) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10004 is 1
RuntimeLogger 2025-03-24 11:03:38,233 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5727, label: label_561988f831964eb0_a11e1c4d694f2861, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814218178, commit time: 1742814218229, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:38,234 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5727, db: 10162
RuntimeLogger 2025-03-24 11:03:38,248 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3716, visibleVersionTime: 1742814218245
RuntimeLogger 2025-03-24 11:03:38,252 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33536, eventTime=1742814218251, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:38,252 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5727, label: label_561988f831964eb0_a11e1c4d694f2861, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814218178, commit time: 1742814218229, finish time: 1742814218245, reason: successfully, publish times 1, publish result QUORUM_SUCC
RuntimeLogger 2025-03-24 11:03:38,457 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5728 with label label_75cca83f69cc446d_90351dea77f31c53 from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:38,458 INFO (mysql-nio-pool-44|6692) [OlapInsertExecutor.beforeExec():197] start insert [label_75cca83f69cc446d_90351dea77f31c53] with query id 75cca83f69cc446d-90351dea77f31c53 and txn id 5728
RuntimeLogger 2025-03-24 11:03:38,458 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():776] dispatch load job: 75cca83f69cc446d-90351dea77f31c53 to [TNetworkAddress(hostname:172.20.80.7, port:9060)]
RuntimeLogger 2025-03-24 11:03:38,458 INFO (mysql-nio-pool-44|6692) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10005 is 1
RuntimeLogger 2025-03-24 11:03:38,515 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5728, label: label_75cca83f69cc446d_90351dea77f31c53, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814218457, commit time: 1742814218511, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:38,516 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5728, db: 10162
RuntimeLogger 2025-03-24 11:03:38,531 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3717, visibleVersionTime: 1742814218527
RuntimeLogger 2025-03-24 11:03:38,534 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33538, eventTime=1742814218534, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:38,534 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5728, label: label_75cca83f69cc446d_90351dea77f31c53, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814218457, commit time: 1742814218511, finish time: 1742814218527, reason: successfully, publish times 1, publish result QUORUM_SUCC
RuntimeLogger 2025-03-24 11:03:38,627 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.beginTransaction():397] begin transaction: txn id 5729 with label label_a40636a55e69454d_aaeadc94bcf6042b from coordinator FE: 172.20.80.22, listener id: -1
RuntimeLogger 2025-03-24 11:03:38,628 INFO (mysql-nio-pool-44|6692) [OlapInsertExecutor.beforeExec():197] start insert [label_a40636a55e69454d_aaeadc94bcf6042b] with query id a40636a55e69454d-aaeadc94bcf6042b and txn id 5729
RuntimeLogger 2025-03-24 11:03:38,628 INFO (mysql-nio-pool-44|6692) [Coordinator.execInternal():776] dispatch load job: a40636a55e69454d-aaeadc94bcf6042b to [TNetworkAddress(hostname:172.20.80.6, port:9060)]
RuntimeLogger 2025-03-24 11:03:38,629 INFO (mysql-nio-pool-44|6692) [Coordinator.sendPipelineCtx():1038] num local sink for backend 10006 is 1
RuntimeLogger 2025-03-24 11:03:38,685 INFO (mysql-nio-pool-44|6692) [DatabaseTransactionMgr.commitTransaction():828] transaction:[TransactionState. transaction id: 5729, label: label_a40636a55e69454d_aaeadc94bcf6042b, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1742814218627, commit time: 1742814218680, finish time: -1, reason: ] successfully committed
RuntimeLogger 2025-03-24 11:03:38,688 INFO (PUBLISH_VERSION|20) [PublishVersionDaemon.genPublishTask():178] send publish tasks for transaction: 5729, db: 10162
RuntimeLogger 2025-03-24 11:03:38,702 INFO (PUBLISH_VERSION|20) [OlapTable.updateVisibleVersionAndTime():2984] updateVisibleVersionAndTime, tableName: audit_operation_logs, visibleVersion, 3718, visibleVersionTime: 1742814218698
RuntimeLogger 2025-03-24 11:03:38,706 INFO (PUBLISH_VERSION|20) [MTMVService.processEvent():185] processEvent, Event: TableEvent{ctlId=0, ctlName='internal', dbId=10162, dbName='doris_db', tableId=11468, tableName='audit_operation_logs'} Event{eventId=33540, eventTime=1742814218705, eventType=DATA_CHANGE}
RuntimeLogger 2025-03-24 11:03:38,706 INFO (PUBLISH_VERSION|20) [DatabaseTransactionMgr.finishTransaction():1110] finish transaction TransactionState. transaction id: 5729, label: label_a40636a55e69454d_aaeadc94bcf6042b, db id: 10162, table id list: 11468, callback id: -1, coordinator: FE: 172.20.80.22, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1742814218627, commit time: 1742814218680, finish time: 1742814218698, reason: successfully, publish times 1, publish result QUORUM_SUCC
be.gc.log 崩溃前日志
OpenJDK 64-Bit Server VM (25.352-b11) for linux-aarch64 JRE (1.8.0_352-b11), built on Nov 10 2022 10:27:39 by "root" with gcc 7.3.0
Memory: 64k page, physical 9437184k(9070720k free), swap 0k(0k free)
CommandLine flags: -XX:-CriticalJNINatives -XX:InitialHeapSize=150994944 -XX:MaxHeapSize=2147483648 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
1.651: [GC (Allocation Failure) 40960K->6064K(157184K), 0.0145441 secs]
2.031: [GC (Allocation Failure) 47024K->4005K(157184K), 0.0146940 secs]
2.659: [GC (Metadata GC Threshold) 41592K->11638K(157184K), 0.0130492 secs]
2.673: [Full GC (Metadata GC Threshold) 11638K->11254K(125440K), 0.0588623 secs]
2.980: [GC (Allocation Failure) 52214K->14154K(141824K), 0.0038371 secs]
3.106: [GC (Allocation Failure) 75082K->11685K(162816K), 0.0038224 secs]
3.232: [GC (Allocation Failure) 90021K->11454K(182784K), 0.0035067 secs]
3.337: [GC (Allocation Failure) 103102K->11454K(182784K), 0.0034706 secs]
3.441: [GC (Allocation Failure) 103102K->11422K(231936K), 0.0032081 secs]
3.643: [GC (Allocation Failure) 152734K->11568K(231424K), 0.0026882 secs]
3.797: [GC (Allocation Failure) 152880K->11488K(271360K), 0.0027893 secs]
be.warning.log 崩溃前日志
RuntimeLogger W20250324 19:03:26.494997 1482 status.h:413] meet error status: [TIMEOUT]Query tiemout
0# doris::ResultBufferMgr::cancel_thread()
1# doris::Thread::supervise_thread(void*)
2# ?
3# ?
RuntimeLogger W20250324 19:03:27.495492 1482 status.h:413] meet error status: [TIMEOUT]Query tiemout
0# doris::ResultBufferMgr::cancel_thread()
1# doris::Thread::supervise_thread(void*)
2# ?
3# ?
RuntimeLogger W20250324 19:03:27.495751 1482 status.h:413] meet error status: [TIMEOUT]Query tiemout
0# doris::ResultBufferMgr::cancel_thread()
1# doris::Thread::supervise_thread(void*)
2# ?
3# ?
RuntimeLogger W20250324 19:03:33.496768 1482 status.h:413] meet error status: [TIMEOUT]Query tiemout
0# doris::ResultBufferMgr::cancel_thread()
1# doris::Thread::supervise_thread(void*)
2# ?
3# ?
RuntimeLogger W20250324 19:03:43.498467 1482 status.h:413] meet error status: [TIMEOUT]Query tiemout
0# doris::ResultBufferMgr::cancel_thread()
1# doris::Thread::supervise_thread(void*)
2# ?
3# ?
be.out
be.put
```log
INFO: java_cmd /usr/lib/jvm/java/bin/java
INFO: jdk_version 8
INFO: java_cmd /usr/lib/jvm/java/bin/java
INFO: jdk_version 8
INFO: java_cmd /usr/lib/jvm/java/bin/java
INFO: jdk_version 8
INFO: java_cmd /usr/lib/jvm/java/bin/java
INFO: jdk_version 8
INFO: java_cmd /usr/lib/jvm/java/bin/java
INFO: jdk_version 8