Doris be mem_limit 参数问题

Viewed 7

Doris版本:2.1.5
问题描述:
Doris be设置mem_limit后不生效,设置了20%,机器内存32G,但是实际使用了12G多。如图:
image.png

2 Answers

mem_limit 这块只是一个软限

感谢回答!
补充一些信息,Doris be进程在被oom killer干掉之前,Memory Summary如下:

W20241115 11:33:25.314775 25264 mem_tracker_limiter.cpp:389]
Process Memory Summary:
os physical memory 31.26 GB. process memory used 12.94 GB(= 13.02 GB[vm/rss] - 85.37 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 6.25 GB, soft limit 5.63 GB. sys available memory 47.05 MB(= 47.05 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 1.56 GB, warning water mark 3.13 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=85.37 MB(89516432 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=183.91 MB(192844000 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum of all trackers, Type=overview, Limit=-1.00 B(-1 B), Used=269.28 MB(282360432 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process resident memory, Type=overview, Limit=-1.00 B(-1 B), Used=13.02 GB(13982605312 B), Peak=13.03 GB(13994246144 B)
MemTrackerLimiter Label=reserve_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=process virtual memory, Type=overview, Limit=-1.00 B(-1 B), Used=34.42 GB(36954378240 B), Peak=34.42 GB(36954382336 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=113.71 MB(119235865 B), Peak=113.71 MB(119235865 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=47.14 MB(49432320 B), Peak=47.14 MB(49432320 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=0(0 B)
看了官方提供的排查oom的思路:
https://doris.apache.org/zh-CN/docs/admin-manual/memory-management/memory-issue-faq
从MemTracker信息上看jemalloc不高, virtual memory挺高的
image.png
还有什么手段定位内存占用高的原因吗?