2.1.5 三个be, 只有一个be内存占用过高，会到达90%+ 。另外两个50%-60%

Viewed 77
resource-admin 2.1 cluster

问题出现在 新增了两个大的实时任务后 streamload
be.info 里的 memtracker 日志

I20250106 09:24:06.650542 23921 cache_manager.h:78] [MemoryGC] cache no prune stale, last prune less than interval 10, now 1736126646, last timestamp 1736126641
I20250106 09:24:06.650705 23921 memory_reclamation.cpp:239] [MemoryGC] start GC work load group that enable overcommit, number of group: 1, request_free_memory:12151774126, total_exceeded_memory:30109534756, request less than exceeded, try free size = ((group used - group limit) / all group total_exceeded_memory) * request_free_memory.
I20250106 09:24:06.650759 23921 workload_group.cpp:192] [MemoryGC] work load group start gc, id:1 name:normal, memory limit: 36455322378, used: 66565014830, need_free_mem: 12151774126.

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=1.29 GB(1382316898 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=61.60 GB(66137857813 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=102.68 MB(107665040 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=535.76 MB(561787019 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=33.40 GB(35866261160 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2752932752 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=99.47 GB(106808820682 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=108.15 GB(116123525120 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1675.10 GB(1798622793728 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844380815 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=6.02 GB(6463518851 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.47 GB(1573740544 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.59 GB(1708596841 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=152.74 KB(156408 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=662.16 MB(694324803 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=182.36 KB(186738 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18329040 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.26 KB(91401 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=26.98 MB(28286695 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=70.61 MB(74039296 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=3.72 GB(3990921838 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=162.09 MB(169961767 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=178.38 MB(187040624 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=21.73 GB(23336383951 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=19.47 GB(20902049787 B), Peak=19.62 GB(21066623474 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=14.73 GB(15816608630 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=5.28 GB(5673793337 B), Peak=6.21 GB(6671406955 B)
MemTrackerLimiter Label=CumulativeCompaction:27568520, Type=compaction, Limit=-1.00 B(-1 B), Used=1.29 GB(1382644598 B), Peak=1.86 GB(2002207956 B)
MemTrackerLimiter Label=Load#Id=b8c37585b97d4e28-b55cf8242d28c3d1, Type=load, Limit=2.00 GB(2147483648 B), Used=319.59 MB(335110932 B), Peak=5.36 GB(5752897497 B)
MemTrackerLimiter Label=(FromLoadChannel)Load#Id=0742a8c4e0b6f8af-35d86b109a2dbeb3, Type=load, Limit=-1.00 B(-1 B), Used=68.81 MB(72156727 B), Peak=68.81 MB(72156727 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=Query#Id=d21b878f99e74e27-aa6db46427ecfc83, Type=query, Limit=2.00 GB(2147483648 B), Used=21.89 MB(22949728 B), Peak=55.03 MB(57698408 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=Query#Id=186628affb244966-b623973784f8a996, Type=query, Limit=2.00 GB(2147483648 B), Used=15.38 MB(16129216 B), Peak=58.53 MB(61376376 B)
MemTrackerLimiter Label=Query#Id=a5cb6ff194ed4b04-81835049aa744876, Type=query, Limit=2.00 GB(2147483648 B), Used=5.64 MB(5911584 B), Peak=49.78 MB(52194440 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Query#Id=4793bd774a474c37-bbd7506dbf8004ab, Type=query, Limit=2.00 GB(2147483648 B), Used=1.63 MB(1707712 B), Peak=1.63 MB(1707712 B)
MemTrackerLimiter Label=Query#Id=fa5caef5f13a49cf-a691aaffde9c8475, Type=query, Limit=2.00 GB(2147483648 B), Used=1.09 MB(1139904 B), Peak=92.70 MB(97206368 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=662.16 MB(694324803 B), Peak=14.00 GB(15029442003 B)
I20250106 09:24:08.929039 23921 cache_manager.h:78] [MemoryGC] cache no prune stale, last prune less than interval 10, now 1736126648, last timestamp 1736126641
I20250106 09:24:08.929193 23921 memory_reclamation.cpp:239] [MemoryGC] start GC work load group that enable overcommit, number of group: 1, request_free_memory:12151774126, total_exceeded_memory:29716277470, request less than exceeded, try free size = ((group used - group limit) / all group total_exceeded_memory) * request_free_memory.
I20250106 09:24:08.929235 23921 workload_group.cpp:192] [MemoryGC] work load group start gc, id:1 name:normal, memory limit: 36455322378, used: 66170754828, need_free_mem: 12151774126.

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=1.32 GB(1413633656 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=52.27 GB(56121837846 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=57.22 MB(59999056 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=993.18 MB(1041420994 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=42.98 GB(46146925608 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2752932752 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=100.15 GB(107536749912 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=107.91 GB(115870261248 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1675.08 GB(1798606008320 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844380857 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=4.37 GB(4695176949 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.39 GB(1492443136 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.44 GB(1545672833 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=153.35 KB(157033 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=693.61 MB(727297825 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=182.36 KB(186738 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18329040 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.37 KB(91510 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=26.97 MB(28275113 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=71.02 MB(74465280 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=4.16 GB(4465455362 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=166.28 MB(174353589 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=178.64 MB(187312691 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=16.53 GB(17746859433 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=15.59 GB(16734973325 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=14.79 GB(15882046982 B), Peak=19.92 GB(21390846828 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=5.29 GB(5677876551 B), Peak=6.21 GB(6671406955 B)
MemTrackerLimiter Label=CumulativeCompaction:27568520, Type=compaction, Limit=-1.00 B(-1 B), Used=1.32 GB(1413633656 B), Peak=1.86 GB(2002207956 B)
MemTrackerLimiter Label=Load#Id=6c1cabc61d97466d-ba953c45947898c4, Type=load, Limit=2.00 GB(2147483648 B), Used=73.01 MB(76561027 B), Peak=84.33 MB(88425984 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Query#Id=bfeb8f4eb9cd4da1-bf57311e0d3f0dcf, Type=query, Limit=2.00 GB(2147483648 B), Used=168.12 KB(172160 B), Peak=5.59 MB(5861504 B)
MemTrackerLimiter Label=Load#Id=be4e18f7f9ae9b5a-5ba904e6f150ffa8, Type=load, Limit=2.00 GB(2147483648 B), Used=4.12 KB(4224 B), Peak=4.12 KB(4224 B)
MemTrackerLimiter Label=Load#Id=ed4ad4124b1c4224-99ff7e3548c41fa8, Type=load, Limit=2.00 GB(2147483648 B), Used=4.03 KB(4128 B), Peak=4.03 KB(4128 B)
MemTrackerLimiter Label=Load#Id=4c45a85700aded31-e614bacb0eeb79ad, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=a84f79b753bd20d1-c95c2760ee9835ae, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=67489eac97e67597-13e0680d111a1995, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=693.61 MB(727297825 B), Peak=14.00 GB(15029442003 B)
I20250106 09:24:20.496553 23921 cache_manager.h:78] [MemoryGC] cache no prune stale, last prune less than interval 10, now 1736126660, last timestamp 1736126652
I20250106 09:24:20.497186 23921 memory_reclamation.cpp:239] [MemoryGC] start GC work load group that enable overcommit, number of group: 1, request_free_memory:12151774126, total_exceeded_memory:19699447865, request less than exceeded, try free size = ((group used - group limit) / all group total_exceeded_memory) * request_free_memory.
I20250106 09:24:20.497435 23921 workload_group.cpp:192] [MemoryGC] work load group start gc, id:1 name:normal, memory limit: 36455322378, used: 56154785753, need_free_mem: 12151774126.

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=1.34 GB(1440665979 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=45.83 GB(49214934686 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=57.06 MB(59826896 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=587.34 MB(615872971 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=51.37 GB(55155663288 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2752015248 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=101.74 GB(109238979068 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=107.44 GB(115362017280 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1674.84 GB(1798345834496 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844404974 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=2.56 GB(2745708619 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.46 GB(1566793728 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.46 GB(1566612048 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=153.36 KB(157041 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=506.69 MB(531298984 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=182.36 KB(186738 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18329040 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.37 KB(91510 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=28.45 MB(29832032 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=72.41 MB(75923456 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=3.76 GB(4036820091 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=166.80 MB(174897505 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=179.69 MB(188421964 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=14.85 GB(15949364822 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=14.53 GB(15599413541 B), Peak=19.92 GB(21390846828 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=11.15 GB(11973772830 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=5.28 GB(5668407079 B), Peak=6.21 GB(6671406955 B)
MemTrackerLimiter Label=CumulativeCompaction:27568520, Type=compaction, Limit=-1.00 B(-1 B), Used=1.34 GB(1440665979 B), Peak=1.86 GB(2002207956 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=(FromLoadChannel)Load#Id=d7477c3293f365aa-52f59f0a6db959ac, Type=load, Limit=-1.00 B(-1 B), Used=22.88 MB(23987390 B), Peak=45.76 MB(47983939 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Load#Id=a145d2ff3489e763-4d7d69890dae9e96, Type=load, Limit=2.00 GB(2147483648 B), Used=4.12 KB(4224 B), Peak=4.12 KB(4224 B)
MemTrackerLimiter Label=Load#Id=cb46d142f5e295a4-9202ab5e2b6fa6a6, Type=load, Limit=2.00 GB(2147483648 B), Used=4.03 KB(4128 B), Peak=4.03 KB(4128 B)
MemTrackerLimiter Label=Load#Id=8943bb73ae1e07eb-aed9a638c1f0dfbe, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=684d11aa96c8c188-e3c896930bb7268a, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=fb4833ab0611205a-48b17d3e4e0d038b, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=5c44886cae192f4e-64d1415d5e19c697, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=506.69 MB(531298984 B), Peak=14.00 GB(15029442003 B)
I20250106 09:24:27.372817 23921 cache_manager.h:78] [MemoryGC] cache no prune stale, last prune less than interval 10, now 1736126667, last timestamp 1736126663
I20250106 09:24:27.372951 23921 memory_reclamation.cpp:239] [MemoryGC] start GC work load group that enable overcommit, number of group: 1, request_free_memory:12151774126, total_exceeded_memory:12795896545, request less than exceeded, try free size = ((group used - group limit) / all group total_exceeded_memory) * request_free_memory.
I20250106 09:24:27.372997 23921 workload_group.cpp:192] [MemoryGC] work load group start gc, id:1 name:normal, memory limit: 36455322378, used: 49251218923, need_free_mem: 12151774126.

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=1.37 GB(1469915325 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=59.54 GB(63925504155 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=140.13 MB(146940856 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=679.67 MB(712684039 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=37.05 GB(39777357224 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2752932752 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=101.31 GB(108785334351 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=109.03 GB(117065314304 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1675.07 GB(1798589222912 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844404974 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=2.48 GB(2658323382 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.41 GB(1513578496 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.46 GB(1568615686 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=152.86 KB(156533 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=730.45 MB(765933671 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=182.36 KB(186738 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18330480 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.47 KB(91619 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=30.01 MB(31465800 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=72.45 MB(75964416 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=3.85 GB(4130867434 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=167.90 MB(176053178 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=181.19 MB(189989056 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=15.31 GB(16435301245 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=14.86 GB(15959816673 B), Peak=19.92 GB(21390846828 B)
MemTrackerLimiter Label=Load#Id=edc30f25874b4778-873f29663c604522, Type=load, Limit=2.00 GB(2147483648 B), Used=14.11 GB(15155439359 B), Peak=14.11 GB(15155439359 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=9.93 GB(10666008672 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=5.31 GB(5696606246 B), Peak=6.21 GB(6671406955 B)
MemTrackerLimiter Label=CumulativeCompaction:27568520, Type=compaction, Limit=-1.00 B(-1 B), Used=1.37 GB(1469915325 B), Peak=1.86 GB(2002207956 B)
MemTrackerLimiter Label=Query#Id=342298895c8b477f-830397c6f83c1b07, Type=query, Limit=2.00 GB(2147483648 B), Used=39.10 MB(41002600 B), Peak=39.10 MB(41002600 B)
MemTrackerLimiter Label=Query#Id=42f5f0c852df48d8-83bfb05ef3ab098b, Type=query, Limit=2.00 GB(2147483648 B), Used=36.79 MB(38574384 B), Peak=36.79 MB(38574384 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=(FromLoadChannel)Load#Id=d7477c3293f365aa-52f59f0a6db959ac, Type=load, Limit=-1.00 B(-1 B), Used=11.73 MB(12299032 B), Peak=45.76 MB(47983939 B)
MemTrackerLimiter Label=Query#Id=3d044d4d5bb0413c-afd588eed71b890a, Type=query, Limit=2.00 GB(2147483648 B), Used=7.19 MB(7536976 B), Peak=27.15 MB(28467024 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Load#Id=fc4c84ca0dda32b1-5ed45fa273358c86, Type=load, Limit=2.00 GB(2147483648 B), Used=4.12 KB(4224 B), Peak=4.12 KB(4224 B)
MemTrackerLimiter Label=Load#Id=644f8b45e5aa88e6-730b3700435b3ca8, Type=load, Limit=2.00 GB(2147483648 B), Used=4.03 KB(4128 B), Peak=4.03 KB(4128 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=730.45 MB(765933671 B), Peak=14.00 GB(15029442003 B)
I20250106 09:24:32.919101 23920 daemon.cpp:220] os physical memory 125.75 GB. process memory used 73.12 GB(= 109.03 GB[vm/rss] - 35.91 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 113.17 GB, soft limit 101.85 GB. sys available memory 6.08 GB(= 6.08 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.29 GB, warning water mark 12.57 GB.
I20250106 09:24:32.919137 24803 allocator.cpp:81] Query:e95d3088b14975-93a82df37ca0aac0 waiting for enough memory in thread id:139622395717376, maximum 1000ms, Allocator sys memory check failed: Cannot alloc:4194304, consuming tracker:<Load#Id=e95d3088b14975-93a82df37ca0aac0>, peak used 19274434688, current used 16435301245, exec node:, process memory used 73.12 GB exceed limit 113.17 GB or sys available memory 6.08 GB less than low water mark 6.29 GB..
W20250106 09:24:32.924674 22778 status.h:412] meet error status: [INTERNAL_ERROR]Fail to do LZ4F compress due to exception

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=1.33 GB(1428598298 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=59.54 GB(63925504155 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=140.13 MB(146940856 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=679.32 MB(712315399 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=35.91 GB(38558468856 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2752932752 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=100.14 GB(107524760316 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=109.03 GB(117065314304 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1675.07 GB(1798589222912 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844404974 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=2.46 GB(2644560147 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.41 GB(1513578496 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=1.46 GB(1568615686 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=152.86 KB(156533 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=730.45 MB(765933671 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=182.36 KB(186738 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18330480 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.47 KB(91619 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=30.01 MB(31465800 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=72.09 MB(75595776 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=3.85 GB(4130867434 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=167.90 MB(176053178 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=181.19 MB(189989056 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=15.31 GB(16435301245 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=14.86 GB(15959816673 B), Peak=19.92 GB(21390846828 B)
MemTrackerLimiter Label=Load#Id=edc30f25874b4778-873f29663c604522, Type=load, Limit=2.00 GB(2147483648 B), Used=14.11 GB(15155439359 B), Peak=14.11 GB(15155439359 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=9.93 GB(10666008672 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=5.31 GB(5696606246 B), Peak=6.21 GB(6671406955 B)
MemTrackerLimiter Label=CumulativeCompaction:27568520, Type=compaction, Limit=-1.00 B(-1 B), Used=1.33 GB(1428598298 B), Peak=1.86 GB(2002207956 B)
MemTrackerLimiter Label=Query#Id=342298895c8b477f-830397c6f83c1b07, Type=query, Limit=2.00 GB(2147483648 B), Used=39.10 MB(41002600 B), Peak=39.10 MB(41002600 B)
MemTrackerLimiter Label=Query#Id=42f5f0c852df48d8-83bfb05ef3ab098b, Type=query, Limit=2.00 GB(2147483648 B), Used=36.79 MB(38574384 B), Peak=36.79 MB(38574384 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=(FromLoadChannel)Load#Id=d7477c3293f365aa-52f59f0a6db959ac, Type=load, Limit=-1.00 B(-1 B), Used=11.73 MB(12299032 B), Peak=45.76 MB(47983939 B)
MemTrackerLimiter Label=Query#Id=3d044d4d5bb0413c-afd588eed71b890a, Type=query, Limit=2.00 GB(2147483648 B), Used=7.19 MB(7536976 B), Peak=27.15 MB(28467024 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Load#Id=fc4c84ca0dda32b1-5ed45fa273358c86, Type=load, Limit=2.00 GB(2147483648 B), Used=4.12 KB(4224 B), Peak=4.12 KB(4224 B)
MemTrackerLimiter Label=Load#Id=644f8b45e5aa88e6-730b3700435b3ca8, Type=load, Limit=2.00 GB(2147483648 B), Used=4.03 KB(4128 B), Peak=4.03 KB(4128 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=730.45 MB(765933671 B), Peak=14.00 GB(15029442003 B)
I20250106 09:24:32.963390 23921 lru_cache_policy.h:184] [MemoryGC] MowDeleteBitmapAggCache not need prune all, force is false, LRUCacheType::SIZE consumption 31465800, CACHE_MIN_FREE_SIZE 67108864
I20250106 09:24:32.963433 23921 lru_cache_policy.h:190] [MemoryGC] CreateTabletRRIdxCache not need prune all, force is false, LRUCacheType::NUMBER usage 847, CACHE_MIN_FREE_NUMBER 1024
I20250106 09:24:32.963442 23921 lru_cache_policy.h:172] [MemoryGC] DataPageCache prune all start, consumption 4130867434, usage 4161051657

Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=8.00 MB(8393485 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=1.39 GB(1496000129 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=11.38 MB(11932782 B), Peak=2.60 GB(2792903643 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=24.69 GB(26506753165 B), Peak=95.47 GB(102514823208 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=75.22 MB(78872208 B), Peak=77.77 GB(83509618272 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=-3.31 GB(-3557663658 B), Peak=28.62 GB(30725969884 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=78.93 GB(84754448000 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=2.56 GB(2751610512 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=102.95 GB(110545953009 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=102.80 GB(110384480256 B), Peak=113.79 GB(122184192000 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=1674.61 GB(1798094053376 B), Peak=1675.86 GB(1799438176256 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=-3.58 GB(-3844452798 B), Peak=10.61 MB(11129936 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=62.36 KB(63860 B), Peak=15.78 GB(16943070065 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=1.06 GB(1141506048 B), Peak=7.21 GB(7738073088 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=339.03 MB(355501325 B), Peak=2.34 GB(2516789604 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=116.25 KB(119038 B), Peak=339.50 KB(347649 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=296.47 MB(310866369 B), Peak=14.00 GB(15029442003 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=181.34 KB(185695 B), Peak=195.20 KB(199883 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=17.48 MB(18327600 B), Peak=33.98 MB(35632080 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900833 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=89.47 KB(91619 B), Peak=246.16 KB(252072 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=29.66 MB(31102541 B), Peak=207.54 MB(217618839 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=-15.06 MB(-15790849 B), Peak=316.52 MB(331899751 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=5.64 KB(5778 B), Peak=5.64 KB(5778 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=22.62 MB(23715840 B), Peak=251.91 MB(264142848 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=192.23 MB(201565456 B), Peak=20.30 GB(21795481272 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=11.15 MB(11687539 B), Peak=2.24 GB(2404834345 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=64.29 MB(67417657 B), Peak=9.93 GB(10662281466 B)
MemTrackerLimiter Label=Load#Id=b70b17d6b3b34941-94d23efef97b1d66, Type=load, Limit=2.00 GB(2147483648 B), Used=11.16 GB(11979338656 B), Peak=39.43 GB(42342237936 B)
MemTrackerLimiter Label=Load#Id=483ebeab609b458c-91bd21be906aab63, Type=load, Limit=2.00 GB(2147483648 B), Used=8.33 GB(8942086328 B), Peak=8.51 GB(9134355872 B)
MemTrackerLimiter Label=Load#Id=e95d3088b14975-93a82df37ca0aac0, Type=load, Limit=2.00 GB(2147483648 B), Used=3.54 GB(3799244045 B), Peak=17.95 GB(19274434688 B)
MemTrackerLimiter Label=Load#Id=41b4aad5d6f946d2-90a9d2a7f4ea5046, Type=load, Limit=2.00 GB(2147483648 B), Used=1.66 GB(1783964264 B), Peak=19.92 GB(21390846828 B)
MemTrackerLimiter Label=Query#Id=dcaad6a609b44efa-bc0c76b4129dea50, Type=query, Limit=2.00 GB(2147483648 B), Used=34.12 MB(35778256 B), Peak=40.39 MB(42354256 B)
MemTrackerLimiter Label=Query#Id=4da1626342f84d7d-947ce14b0b56edee, Type=query, Limit=2.00 GB(2147483648 B), Used=18.88 MB(19793696 B), Peak=18.88 MB(19793696 B)
MemTrackerLimiter Label=Query#Id=6877af75f6954b40-ac4ace3dff23f3ad, Type=query, Limit=2.00 GB(2147483648 B), Used=17.75 MB(18615424 B), Peak=32.50 MB(34074808 B)
MemTrackerLimiter Label=CumulativeCompaction:16943414, Type=compaction, Limit=-1.00 B(-1 B), Used=10.94 MB(11476506 B), Peak=22.55 MB(23642451 B)
MemTrackerLimiter Label=Query#Id=19952f8bea0b4234-a8a4dccad29c5f75, Type=query, Limit=2.00 GB(2147483648 B), Used=5.18 MB(5433216 B), Peak=22.82 MB(23931488 B)
MemTrackerLimiter Label=Load#Id=5143e5d9b632074c-d5b9eb65980867b9, Type=load, Limit=2.00 GB(2147483648 B), Used=4.12 KB(4224 B), Peak=4.12 KB(4224 B)
MemTrackerLimiter Label=Load#Id=b24f07edd5bc9c11-f70172fd37ea7f85, Type=load, Limit=2.00 GB(2147483648 B), Used=4.03 KB(4128 B), Peak=4.03 KB(4128 B)
MemTrackerLimiter Label=Load#Id=5642ad020af6c37c-93c1cc1f89ba3383, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=2749e1607c83588f-f49f79e60ae62cb7, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=514c682b3f1a2830-ab3bbcd943fe1ba1, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=4d47da3d988624a0-f97876699b57969c, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=296.47 MB(310866369 B), Peak=14.00 GB(15029442003 B)
I20250106 09:25:17.844852 23921 lru_cache_policy.h:148] [MemoryGC] MowDeleteBitmapAggCache not need prune stale, LRUCacheType::SIZE consumption 31102541 less than CACHE_MIN_FREE_SIZE 67108864
I20250106 09:25:17.844959 23921 lru_cache_policy.h:154] [MemoryGC] CreateTabletRRIdxCache not need prune stale, LRUCacheType::NUMBER usage 847 less than CACHE_MIN_FREE_NUMBER 1024
I20250106 09:25:17.844971 23921 lru_cache_policy.h:135] [MemoryGC] DataPageCache prune stale start, consumption 202118386, usage 204737600

张张张1231
1 Answers

实时任务 会报错的是内存不足的问题。
对于这个问题，只有一个be内存占用过高，会到达90%+ 。另外两个50%-60%，你可以看下你各个BE的tablet分布情况如何？各个be上还有其他的应用吗？

徐振超@SelectDB6796