Doris 2.1.4版本，be内存持续升高问题

Viewed 434
2.1

doris版本信息：doris 版本 2.1.4，Version: doris-2.1.4-rc03-e93678fd1e

以下信息采集自某台内存即将达到阈值的be节点，如果有需要补充的信息，请社区大佬提出，会尽快补充~

  1. Memtracker页面截图
    image.png

  2. be内存监控看板
    image.png

is_cpc1
6 Answers

be节点上是否有混部的情况

张彬华5610

2.1.4 版本，参考这个文档 https://doris.apache.org/zh-CN/docs/admin-manual/memory-management/memory-tracker/，看下 http://beip:8040/mem_tracker?type=global 中 SegmentCache 的值是多少，如果值较大，参考下面的文档关闭 SegmentCache

3.5.2. Label=SegmentCache 内存使用多
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.64 MB(1720543 B), Peak=18.78 MB(19691997 B)

缓存已打开的 Segment 的内存大小，SegmentCache 内存使用多时。

  • 执行 curl -X POST http:/BeIP:8040/api/update_config?disable_segment_cache=true 对正在运行的 BE 清空 SegmentCache 并禁用，等待最长 10 分钟后生效，但这是临时方法，BE 重启后 SegmentCache 将重新生效。
  • 若经过上一步操作后确认要减少 SegmentCache 的内存使用，参考 BE 配置项，在 conf/be.conf 中调整 segment_cache_capacity 或 segment_cache_memory_percentage 减小 SegmentCache 的容量，或调小 tablet_rowset_stale_sweep_time_sec 减小 DataPageCache 缓存有效时长，或者在 conf/be.conf 中增加 disable_segment_cache=true 禁用 SegmentCache 并重启 BE 进程。
xinyiZzz337

内存升高以后，辛苦执行curl http://{be_host}:{be_web_server_port}/api/running_pipeline_tasks/60 拿一下接口返回数据我这边分析下

gabriel17

http://beip:8040/mem_tracker?type=global
image.png

/api/running_pipeline_tasks/60 部分结果返回

128 pipeline fragment contexts are still running! duration_limit=60
No.0 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6d4) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.845985733s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846031495s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846039509s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846071187s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846078438s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846099711s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31

No.1 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6cf) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846112973s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846125922s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846130267s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846142661s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846147307s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846161189s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31

No.2 (elapse_second=16985.845970299s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6c3) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846174531s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6c3
PipelineTask[this = 0x7f79278ac400, state = FINISHED, dry run = false, elapse time = 16985.846187141s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278ac800, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.84619137s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6cf
PipelineTask[this = 0x7f79278c1000, state = FINISHED, dry run = false, elapse time = 16985.846203902s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f7927a39000, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.846208623s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=30, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 1, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=31, parallel_tasks=3, short_circuit_for_probe: 0
    DATA_STREAM_SINK_OPERATOR: id=32, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=30, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=31, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
2.       ExchangeSinkQueueDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
4.           LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             LocalExchangeChannelDependency: id=32, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=32, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d4
PipelineTask[this = 0x7f791c60d800, state = FINISHED, dry run = false, elapse time = 16985.846221462s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=17, parallel_tasks=3, Info: (_num_senders = 1, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=31

No.3 (elapse_second=16985.849338834s, query_timeout_second=-1s, instance_id=7d4ef759b81541cc-a6188f8645bca6e3) : PipelineXFragmentContext Info:
Tasks in instance 0:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f792ff6c400, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849604164s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
    HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
      HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
        DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2.       HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
4.           ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f792ff6c800, state = FINISHED, dry run = false, elapse time = 16985.849630428s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f7930052c00, state = FINISHED, dry run = false, elapse time = 16985.849636096s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6d9
PipelineTask[this = 0x7f7930054000, state = FINISHED, dry run = false, elapse time = 16985.84964211s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=33
Tasks in instance 1:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930054c00, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849649449s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
    HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
      HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
        DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2.       HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
4.           ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930065000, state = FINISHED, dry run = false, elapse time = 16985.849671899s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930133000, state = FINISHED, dry run = false, elapse time = 16985.849676766s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6de
PipelineTask[this = 0x7f7930133c00, state = FINISHED, dry run = false, elapse time = 16985.849681939s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=33
Tasks in instance 2:
Task 0: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f793015a400, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 16985.849687282s], block dependency = SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
EXCHANGE_OPERATOR: id=32, parallel_tasks=3, Info: (_num_senders = 20, _is_merging = false), Queues: (No. 0 queue: (_num_remaining_senders = 19, block_queue size = 0))
  HASH_JOIN_OPERATOR: id=33, parallel_tasks=3, short_circuit_for_probe: 0
    HASH_JOIN_OPERATOR: id=34, parallel_tasks=3, short_circuit_for_probe: 0
      HASH_JOIN_OPERATOR: id=35, parallel_tasks=3, short_circuit_for_probe: 0
        DATA_STREAM_SINK_OPERATOR: id=36, Sink Buffer: (_should_stop = false, _busy_channels = 0, _is_finishing = false), _reach_limit: false0.   SHUFFLE_DATA_DEPENDENCY: id=32, block task = 1, ready=false, _always_ready=false, is cancelled=false
1.     HASH_JOIN_OPERATOR_DEPENDENCY: id=33, block task = 0, ready=true, _always_ready=false, is cancelled=false
2.       HASH_JOIN_OPERATOR_DEPENDENCY: id=34, block task = 0, ready=true, _always_ready=false, is cancelled=false
3.         HASH_JOIN_OPERATOR_DEPENDENCY: id=35, block task = 0, ready=true, _always_ready=false, is cancelled=false
Write Dependency Information: 
4.           ExchangeSinkQueueDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false
5.             BroadcastDependency: id=36, block task = 0, ready=true, _always_ready=false, is cancelled=false

Runtime Filter Dependency Information: 
Finish Dependency Information: 
6.   DATA_STREAM_SINK_OPERATOR_FINISH_DEPENDENCY: id=36, block task = 0, ready=false, _always_ready=false, is cancelled=false

Task 1: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f793015a800, state = FINISHED, dry run = false, elapse time = 16985.849708684s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=8, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=35
Task 2: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f7927803000, state = FINISHED, dry run = false, elapse time = 16985.849713443s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=13, parallel_tasks=3, Info: (_num_senders = 120, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=34
Task 3: QueryId: 7d4ef759b81541cc-a6188f8645bca423
InstanceId: 7d4ef759b81541cc-a6188f8645bca6e3
PipelineTask[this = 0x7f792787a400, state = FINISHED, dry run = false, elapse time = 16985.849718385s], block dependency = NULL, is running = false
operators: 
EXCHANGE_OPERATOR: id=15, parallel_tasks=3, Info: (_num_senders = 5, _is_merging = false)
  HASH_JOIN_SINK_OPERATOR: id=33
is_cpc1

我们使用ROUTINE LOAD大批量导入数据，数据量突然增大时出现内存升高不下降的情况 ，64G内存使用率在90%以上，
SegmentCache使用情况
SegmentCache[size] DetailsTrackerSet none 412640808 393M,537K 862877886 822M,926K
/api/running_pipeline_tasks/60 结果返回
11 pipeline fragment contexts are still running! duration_limit=60
No.0 (elapse_second=1020.716513787s, query_timeout_second=80s, instance_id=3a03972752054c4c-b4cfdce8566dedcc) : PipelineFragmentContext Info: QueryId = 3a03972752054c4c-b4cfdce8566dedcb

No.1 (elapse_second=1062.869769193s, query_timeout_second=80s, instance_id=d5edf15d37e74bcb-b5e7af2c6ffa4b1d) : PipelineFragmentContext Info: QueryId = d5edf15d37e74bcb-b5e7af2c6ffa4b1c

No.2 (elapse_second=1069.47870673s, query_timeout_second=40s, instance_id=738f43293bcf4e61-bf5d0c73b08c1ec6) : PipelineFragmentContext Info: QueryId = 738f43293bcf4e61-bf5d0c73b08c1ec5

No.3 (elapse_second=1140.982039477s, query_timeout_second=40s, instance_id=132709a72d74424e-b56475bd3c742c09) : PipelineFragmentContext Info: QueryId = 132709a72d74424e-b56475bd3c742c08

No.4 (elapse_second=1149.184799515s, query_timeout_second=40s, instance_id=5e8a5f84fcb345b5-86b6877566f230c5) : PipelineFragmentContext Info: QueryId = 5e8a5f84fcb345b5-86b6877566f230c4

No.5 (elapse_second=1156.812229928s, query_timeout_second=80s, instance_id=b209bfc287da40c4-b9382b7b985f77c6) : PipelineFragmentContext Info: QueryId = b209bfc287da40c4-b9382b7b985f77c5

No.6 (elapse_second=1172.23343905s, query_timeout_second=40s, instance_id=50c8533c77ef4869-9f4f36235a374317) : PipelineFragmentContext Info: QueryId = 50c8533c77ef4869-9f4f36235a374316

No.7 (elapse_second=1176.359399118s, query_timeout_second=40s, instance_id=8fb1fb01781b4256-9762b6af93625ab1) : PipelineFragmentContext Info: QueryId = 8fb1fb01781b4256-9762b6af93625ab0

No.8 (elapse_second=1224.287007106s, query_timeout_second=40s, instance_id=3c52ec60a0224718-826cdc95c158f22c) : PipelineFragmentContext Info: QueryId = 3c52ec60a0224718-826cdc95c158f22b

No.9 (elapse_second=1355.246452923s, query_timeout_second=40s, instance_id=29f556481565486d-9ee57426e5456a82) : PipelineFragmentContext Info: QueryId = 29f556481565486d-9ee57426e5456a81

No.10 (elapse_second=1351.626916392s, query_timeout_second=40s, instance_id=a7253fde332941fa-aa89f158ffde01d8) : PipelineFragmentContext Info: QueryId = a7253fde332941fa-aa89f158ffde01d7
这种情况需要关闭 SegmentCache？SegmentCache超过多少的情况下需要关闭？

gfadweeq3

目前看所有2.1.5 的版本的用户，不再出现内存一直升高的问题了。

____226