doris版本信息:doris 版本 2.1.4,Version: doris-2.1.4-rc03-e93678fd1e
以下信息采集自某台内存即将达到阈值的be节点,如果有需要补充的信息,请社区大佬提出,会尽快补充~
-
Memtracker页面截图
-
be内存监控看板
doris版本信息:doris 版本 2.1.4,Version: doris-2.1.4-rc03-e93678fd1e
以下信息采集自某台内存即将达到阈值的be节点,如果有需要补充的信息,请社区大佬提出,会尽快补充~
Memtracker页面截图
be内存监控看板
2.1.4 版本,参考这个文档 https://doris.apache.org/zh-CN/docs/admin-manual/memory-management/memory-tracker/,看下 http://beip:8040/mem_tracker?type=global 中 SegmentCache 的值是多少,如果值较大,参考下面的文档关闭 SegmentCache
Label=SegmentCache
内存使用多MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.64 MB(1720543 B), Peak=18.78 MB(19691997 B)
缓存已打开的 Segment 的内存大小,SegmentCache
内存使用多时。
curl -X POST http:/BeIP:8040/api/update_config?disable_segment_cache=true
对正在运行的 BE 清空 SegmentCache
并禁用,等待最长 10 分钟后生效,但这是临时方法,BE 重启后 SegmentCache
将重新生效。SegmentCache
的内存使用,参考 BE 配置项,在 conf/be.conf
中调整 segment_cache_capacity
或 segment_cache_memory_percentage
减小 SegmentCache
的容量,或调小 tablet_rowset_stale_sweep_time_sec
减小 DataPageCache
缓存有效时长,或者在 conf/be.conf
中增加 disable_segment_cache=true
禁用 SegmentCache
并重启 BE 进程。内存升高以后,辛苦执行curl http://{be_host}:{be_web_server_port}/api/running_pipeline_tasks/60 拿一下接口返回数据我这边分析下
http://beip:8040/mem_tracker?type=global
/api/running_pipeline_tasks/60 部分结果返回
128 pipeline fragment contexts are still running! duration_limit=60
No.0 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6d4) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.845985733s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846031495s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846039509s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846071187s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846078438s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846099711s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
No.1 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6cf) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846112973s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846125922s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846130267s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846142661s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846147307s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846161189s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
No.2 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6c3) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846174531s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846187141s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.84619137s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846203902s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846208623s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
2. ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846221462s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=31
No.3 (elapse_second=16985.849338834s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6e3) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f792ff6c400, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849604164s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2. HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
4. ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f792ff6c800, state = FINISHED, dry run = false, elapse time = 16985.849630428s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f7930052c00, state = FINISHED, dry run = false, elapse time = 16985.849636096s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f7930054000, state = FINISHED, dry run = false, elapse time = 16985.84964211s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=33
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930054c00, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849649449s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2. HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
4. ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930065000, state = FINISHED, dry run = false, elapse time = 16985.849671899s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930133000, state = FINISHED, dry run = false, elapse time = 16985.849676766s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930133c00, state = FINISHED, dry run = false, elapse time = 16985.849681939s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=33
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f793015a400, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849687282s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators:
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0. SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1. HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2. HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3. HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information:
4. ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5. BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
Runtime Filter Dependency Information:
Finish Dependency Information:
6. DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false
Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f793015a800, state = FINISHED, dry run = false, elapse time = 16985.849708684s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f7927803000, state = FINISHED, dry run = false, elapse time = 16985.849713443s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f792787a400, state = FINISHED, dry run = false, elapse time = 16985.849718385s], block dependency = NULL, is running = false
operators:
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
HASH_JOIN_SINK_OPERATOR: id=33
我们使用ROUTINE LOAD大批量导入数据,数据量突然增大时出现内存升高不下降的情况 ,64G内存使用率在90%以上,
SegmentCache使用情况
SegmentCache[size] DetailsTrackerSet none 412640808 393M,537K 862877886 822M,926K
/api/running_pipeline_tasks/60 结果返回
11 pipeline fragment contexts are still running! duration_limit=60
No.0 (elapse_second=1020.716513787s, query_timeout_second=80s, instance_id=3a03972752054c4c-b4cfdce8566dedcc) : PipelineFragmentContext Info: QueryId = 3a03972752054c4c-b4cfdce8566dedcb
No.1 (elapse_second=1062.869769193s, query_timeout_second=80s, instance_id=d5edf15d37e74bcb-b5e7af2c6ffa4b1d) : PipelineFragmentContext Info: QueryId = d5edf15d37e74bcb-b5e7af2c6ffa4b1c
No.2 (elapse_second=1069.47870673s, query_timeout_second=40s, instance_id=738f43293bcf4e61-bf5d0c73b08c1ec6) : PipelineFragmentContext Info: QueryId = 738f43293bcf4e61-bf5d0c73b08c1ec5
No.3 (elapse_second=1140.982039477s, query_timeout_second=40s, instance_id=132709a72d74424e-b56475bd3c742c09) : PipelineFragmentContext Info: QueryId = 132709a72d74424e-b56475bd3c742c08
No.4 (elapse_second=1149.184799515s, query_timeout_second=40s, instance_id=5e8a5f84fcb345b5-86b6877566f230c5) : PipelineFragmentContext Info: QueryId = 5e8a5f84fcb345b5-86b6877566f230c4
No.5 (elapse_second=1156.812229928s, query_timeout_second=80s, instance_id=b209bfc287da40c4-b9382b7b985f77c6) : PipelineFragmentContext Info: QueryId = b209bfc287da40c4-b9382b7b985f77c5
No.6 (elapse_second=1172.23343905s, query_timeout_second=40s, instance_id=50c8533c77ef4869-9f4f36235a374317) : PipelineFragmentContext Info: QueryId = 50c8533c77ef4869-9f4f36235a374316
No.7 (elapse_second=1176.359399118s, query_timeout_second=40s, instance_id=8fb1fb01781b4256-9762b6af93625ab1) : PipelineFragmentContext Info: QueryId = 8fb1fb01781b4256-9762b6af93625ab0
No.8 (elapse_second=1224.287007106s, query_timeout_second=40s, instance_id=3c52ec60a0224718-826cdc95c158f22c) : PipelineFragmentContext Info: QueryId = 3c52ec60a0224718-826cdc95c158f22b
No.9 (elapse_second=1355.246452923s, query_timeout_second=40s, instance_id=29f556481565486d-9ee57426e5456a82) : PipelineFragmentContext Info: QueryId = 29f556481565486d-9ee57426e5456a81
No.10 (elapse_second=1351.626916392s, query_timeout_second=40s, instance_id=a7253fde332941fa-aa89f158ffde01d8) : PipelineFragmentContext Info: QueryId = a7253fde332941fa-aa89f158ffde01d7
这种情况需要关闭 SegmentCache?SegmentCache超过多少的情况下需要关闭?