问题:BE 内存占用很高,重启之后一样很高
背景描述:
业务上尝试利用Doris3 搭建日志分析平台,方案参考官方说明 https://doris.apache.org/zh-CN/docs/gettingStarted/tutorials/log-storage-analysis;其中be配置variant_max_merged_tablet_schema_size = 20480;最近遇到问题BE节点内存打满,根据官方内存分析文档进行了以下操作后依然没有解决
1、BE、集群重启
2、索引删除、表删除
目前排查到的信息
mem_tracker:
overview tc/jemalloc_cache none 87870880 83M,819K -1 -1K
overview tc/jemalloc_metadata none 338331840 322M,674K -1 -1K
overview reserved_memory none 0 0K -1 -1K
overview sum_of_all_trackers none 426202720 406M,469K -1 -1K
overview VmRSS(process resident memory) none 29665964032 27G,643M,684K 29684961280 27G,661M,804K
overview VmSize(process virtual memory) none 48473010176 45G,147M,476K 48473010176 45G,147M,476K
memz:
Allocated: 29871485008, active: 30055370752, metadata: 338397376 (n_thp 151), resident: 30368620544, mapped: 30395469824, retained: 4123127808
vars:
doris_ordinal_index_memory_bytes 目前降低到了0,之前超过了TB
集群配置
3be 32G 8核
3fe 32G 8核
Doris 版本:3.0.2
BE 日志
Process Memory Summary:
os physical memory 30.46 GB. process memory used 26.91 GB(= 27.18 GB[vm/rss] - 270.91 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 27.41 GB, soft limit 24.67 GB. sys available memory 2.56 GB(= 2.56 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 1.52 GB, warning water mark 3.05 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=4.56 KB(4672 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=2.00 MB(2096595 B), Peak=2.00 MB(2096595 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=270.91 MB(284067064 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=330.75 MB(346812400 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=603.65 MB(632976059 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=27.18 GB(29181063168 B), Peak=28.27 GB(30350307328 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=48.19 GB(51739803648 B), Peak=48.19 GB(51739803648 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=2.06 MB(2162688 B), Peak=2.37 MB(2482176 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=103.35 MB(108373824 B), Peak=112.02 MB(117458752 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=4.01 GB(4307306120 B), Peak=4.04 GB(4337515439 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=27.87 MB(29225472 B), Peak=34.58 MB(36264192 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=1.99 MB(2088960 B), Peak=2.05 MB(2154496 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=StreamLoadPipe, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=7.46 KB(7635 B), Peak=7.46 KB(7635 B) MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=EngineIndexChangeTask#tabletId=78239, Type=schema_change, Limit=4.93 GB(5298320719 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=EngineIndexChangeTask#tabletId=78236, Type=schema_change, Limit=4.93 GB(5298320719 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=EngineIndexChangeTask#tabletId=78272, Type=schema_change, Limit=4.93 GB(5298320719 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SnapshotManager, Type=other, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=0(0 B)