一、现象
● 报警详情: MEM usage above 90% (current value: 0.9731329333728482) ● 堆外metaspace内存占用高达3GB多 ● 机器内存耗尽，宕机
二、概念
元空间是jdk1.8开始取代永久代的内存模型,被jvm使用受操作系统管辖的直接内存区域。
jdk1.7内存结构：
jdk1.8内存结构:
三、排查思路
● metaspace增长是逐渐增多，增长速度不均匀，考虑是接口调用或mq ● metaspace主要存放类信息，所以主要怀疑:动态类生成类库的使用 ● 项目中使用的第三方库：fastjson,dubbo,aspectj,spring ● 首先排除了fastjson,dubbo,spring，因为足够稳定和靠谱 ● 但是预发环境和测试环境，依旧无法重现问题
四、排查过程
● jmap查看JVM对象信息,发现大量和反射相关对象被生成
代码语言：javascript复制admin@ip-xxx:~/logs/xxx$ jmap -histo pid
num #instances #bytes class name
----------------------------------------------
1: 836613 102185264 [C
2: 128213 30887816 [B
3: 546707 13120968 java.lang.String
4: 390652 12500864 java.util.HashMap$Node
5: 58502 10908984 [I
6: 121161 10662168 java.lang.reflect.Method
7: 149994 10117912 [Ljava.lang.Object;
8: 61139 6432184 [Ljava.util.HashMap$Node;
9: 160882 5148224 java.util.concurrent.ConcurrentHashMap$Node
10: 172682 3822600 [Ljava.lang.Class;
11: 58938 2829024 java.util.HashMap
12: 24263 2688136 java.lang.Class
13: 110260 2646240 java.util.ArrayList
14: 65896 2635840 java.util.LinkedHashMap$Entry
15: 41654 2252544 [Ljava.lang.String;
16: 2459 1894392 [Ljava.util.concurrent.ConcurrentHashMap$Node;
17: 24752 1782144 java.lang.reflect.Field
18: 73982 1775568 java.lang.StringBuilder
19: 29437 1648472 java.util.LinkedHashMap
20: 51507 1648224 java.lang.ref.WeakReference
21: 34020 1632960 org.aspectj.weaver.reflect.ShadowMatchImpl
22: 30742 1229680 com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl$1
23: 76420 1222720 java.lang.Integer
24: 34020 1088640 org.aspectj.weaver.patterns.ExposedState
25: 12658 1012640 java.lang.reflect.Constructor
26: 24377 975080 java.util.HashMap$KeyIterator
27: 59090 945440 java.lang.Object
28: 23290 931600 java.lang.ref.SoftReference
29: 18518 919184 [Ljava.lang.reflect.Method;
30: 35328 847872 com.dianping.cat.internal.shaded.io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry
31: 24738 791616 com.sun.xml.internal.stream.events.CharacterEvent
32: 13612 762272 sun.nio.cs.UTF_8$Encoder
33: 31471 755304 io.termd.core.term.Feature
34: 23095 739040 com.sun.org.apache.xerces.internal.xni.QName
35: 10412 666368 java.util.regex.Matcher
36: 11898 666288 java.beans.MethodDescriptor
● 增加jvm启动参数：-XX: TraceClassLoading -XX: TraceClassUnloading ● 查看项目的类加载日志
代码语言：javascript复制[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_PushArgs from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/admin/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundPushArgs from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_3_PushRefundLine from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_InboxBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_RefundBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_2_RefundLineBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
[Loaded com.alibaba.fastjson.serializer.ASMSerializer_1_PushBO from jar:file:/home/.../-server-prod.jar!/BOOT-INF/lib/fastjson-1.2.58.jar!/]
xxx 03:29:32.045 [http-nio-8888-exec-7] INFO xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:48.813 [http-nio-8888-exec-5] INFO xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:29:56.844 [http-nio-8888-exec-2] INFO xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
xxx 03:30:03.787 [http-nio-8888-exec-9] INFO xxx.proxy.push.PushProxy:[80] - SendMixPush, url: xxx, body: {"message":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx},"sns":{"args":{"order_no":"xxx","return_id":"xxx"},"cellphone":"xxx","country":"India","request_type":"xxx","user_id":xxx}}
● 查看具体代码
代码语言：javascript复制public static String toSneakJSONString(Object object){
SerializeConfig serializeConfig = new SerializeConfig();
serializeConfig.setPropertyNamingStrategy(PropertyNamingStrategy.SnakeCase);
return JSONObject.toJSONString(object, serializeConfig, SerializerFeature.DisableCircularReferenceDetect);
}
● fastjson 官方建议
五、问题根源和解决方案
1.问题根源分析
SerializeConfig 默认会激活 asm，在序列化对象时会为对象生成代理类，然后通过执行代理进行序列化操作，通过这样优化来提高执行性能，但在应用不合理每次新创建 config 的时候就会导致大量生成代码类反而拖慢性能。反序列化时的 ParserConfig 也是同理。
在 jdk8 之前这些代理类会充满 Perm 区导致 FullGC，浪费点 CPU 也不会有大问题，但在 JDK8 中，这些类会大量创建直至充满物理机内存，操作系统检测到该进程是危险进程,出于自我保护机制,进而导致进程被系统杀掉。
2.解决方案
● 代码维度,将SerializeConfig缓存起来,或者使用SerializeConfig.globalInstance全局配置,避免请求维度动态生成类并加载到元空间 ● 调整jvm启动参数,限制metaspace最大内存-XX:MaxMetaspaceSize,宁可FGC或者OOM也不要进程被杀
参考
fastjosn官方issues1
fastjson官方issues2
什么是元空间Metaspace
fastjson中用到的ASM
asm官网
