byte-buddy icon indicating copy to clipboard operation
byte-buddy copied to clipboard

Published 20 hours ago • verified publisher icon raphw

Bytebuddy peg has memory error, how can I solve this problem

Open 1057105012 opened this issue 2 years ago • 5 comments

The project is Spring Gateway, which is developed based on Netty and Reactor, with Pinpoint, Jacoco and self-research agent. When they run together, I get this error, what is the cause There is a point worth paying attention to pinpoint and my agent are bytebuddy development, I Google, seems to be due to concurrent read dependency library caused, do not know how to fix it *** glibc detected *** /wls/apache/springboot/jdk/bin/java: double free or corruption (out): 0x00002ab1b68eb410 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3348e75f4e] /lib64/libc.so.6[0x3348e78cf0] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x644a36)[0x2ab1b152da36] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x65801a)[0x2ab1b154101a] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x46c1c8)[0x2ab1b13551c8] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x46d158)[0x2ab1b1356158] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xa3957c)[0x2ab1b192257c] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x4d9c9a)[0x2ab1b13c2c9a] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x4de6eb)[0x2ab1b13c76eb] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x4debc6)[0x2ab1b13c7bc6] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x4dfa75)[0x2ab1b13c8a75] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xac8765)[0x2ab1b19b1765] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xad4685)[0x2ab1b19bd685] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xad2a5a)[0x2ab1b19bba5a] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xad2dde)[0x2ab1b19bbdde] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0xad3250)[0x2ab1b19bc250] /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so(+0x92a338)[0x2ab1b1813338] /lib64/libpthread.so.0[0x3349207aa1] /lib64/libc.so.6(clone+0x6d)[0x3348ee893d] ======= Memory map: ======== 00400000-00401000 r-xp 00000000 fd:00 785480 /wls/apache/springboot/jdk1.8.0_144/bin/java 00600000-00601000 rw-p 00000000 fd:00 785480 /wls/apache/springboot/jdk1.8.0_144/bin/java 013ef000-01410000 rw-p 00000000 00:00 0 [heap] c0000000-1010ef000 rw-p 00000000 00:00 0 1010ef000-140000000 ---p 00000000 00:00 0 3348a00000-3348a20000 r-xp 00000000 fd:00 654145 /lib64/ld-2.12.so 3348c1f000-3348c20000 r--p 0001f000 fd:00 654145 /lib64/ld-2.12.so 3348c20000-3348c21000 rw-p 00020000 fd:00 654145 /lib64/ld-2.12.so 3348c21000-3348c22000 rw-p 00000000 00:00 0 3348e00000-3348f8a000 r-xp 00000000 fd:00 654148 /lib64/libc-2.12.so 3348f8a000-334918a000 ---p 0018a000 fd:00 654148 /lib64/libc-2.12.so 334918a000-334918e000 r--p 0018a000 fd:00 654148 /lib64/libc-2.12.so 334918e000-334918f000 rw-p 0018e000 fd:00 654148 /lib64/libc-2.12.so 334918f000-3349194000 rw-p 00000000 00:00 0 3349200000-3349217000 r-xp 00000000 fd:00 654155 /lib64/libpthread-2.12.so 3349217000-3349417000 ---p 00017000 fd:00 654155 /lib64/libpthread-2.12.so 3349417000-3349418000 r--p 00017000 fd:00 654155 /lib64/libpthread-2.12.so 3349418000-3349419000 rw-p 00018000 fd:00 654155 /lib64/libpthread-2.12.so 3349419000-334941d000 rw-p 00000000 00:00 0 3349600000-3349602000 r-xp 00000000 fd:00 654156 /lib64/libdl-2.12.so 3349602000-3349802000 ---p 00002000 fd:00 654156 /lib64/libdl-2.12.so 3349802000-3349803000 r--p 00002000 fd:00 654156 /lib64/libdl-2.12.so 3349803000-3349804000 rw-p 00003000 fd:00 654156 /lib64/libdl-2.12.so 3349a00000-3349a07000 r-xp 00000000 fd:00 654174 /lib64/librt-2.12.so 3349a07000-3349c06000 ---p 00007000 fd:00 654174 /lib64/librt-2.12.so 3349c06000-3349c07000 r--p 00006000 fd:00 654174 /lib64/librt-2.12.so 3349c07000-3349c08000 rw-p 00007000 fd:00 654174 /lib64/librt-2.12.so 334a200000-334a283000 r-xp 00000000 fd:00 654159 /lib64/libm-2.12.so 334a283000-334a482000 ---p 00083000 fd:00 654159 /lib64/libm-2.12.so 334a482000-334a483000 r--p 00082000 fd:00 654159 /lib64/libm-2.12.so 334a483000-334a484000 rw-p 00083000 fd:00 654159 /lib64/libm-2.12.so 334ae00000-334ae16000 r-xp 00000000 fd:00 654180 /lib64/libresolv-2.12.so 334ae16000-334b016000 ---p 00016000 fd:00 654180 /lib64/libresolv-2.12.so 334b016000-334b017000 r--p 00016000 fd:00 654180 /lib64/libresolv-2.12.so 334b017000-334b018000 rw-p 00017000 fd:00 654180 /lib64/libresolv-2.12.so 334b018000-334b01a000 rw-p 00000000 00:00 0 3e7d200000-3e7d216000 r-xp 00000000 fd:00 654096 /lib64/libgcc_s-4.4.7-20120601.so.1 3e7d216000-3e7d415000 ---p 00016000 fd:00 654096 /lib64/libgcc_s-4.4.7-20120601.so.1 3e7d415000-3e7d416000 rw-p 00015000 fd:00 654096 /lib64/libgcc_s-4.4.7-20120601.so.1 2ab1b0cc3000-2ab1b0cc4000 rw-p 00000000 00:00 0 2ab1b0cd0000-2ab1b0cd1000 rw-p 00000000 00:00 0 2ab1b0cd1000-2ab1b0ce6000 r-xp 00000000 fd:00 786464 /wls/apache/springboot/jdk1.8.0_144/lib/amd64/jli/libjli.so 2ab1b0ce6000-2ab1b0ee6000 ---p 00015000 fd:00 786464 /wls/apache/springboot/jdk1.8.0_144/lib/amd64/jli/libjli.so 2ab1b0ee6000-2ab1b0ee7000 rw-p 00015000 fd:00 786464 /wls/apache/springboot/jdk1.8.0_144/lib/amd64/jli/libjli.so 2ab1b0ee7000-2ab1b0ee9000 rw-p 00000000 00:00 0 2ab1b0ee9000-2ab1b1bbc000 r-xp 00000000 fd:00 785837 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so 2ab1b1bbc000-2ab1b1dbb000 ---p 00cd3000 fd:00 785837 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so 2ab1b1dbb000-2ab1b1e95000 rw-p 00cd2000 fd:00 785837 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so 2ab1b1e95000-2ab1b1ee0000 rw-p 00000000 00:00 0 2ab1b1ee0000-2ab1b1ee3000 ---p 00000000 00:00 0 2ab1b1ee3000-2ab1b1fe1000 rw-p 00000000 00:00 0 2ab1b1fe1000-2ab1b1fee000 r-xp 00000000 fd:00 785832 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libverify.so 2ab1b1fee000-2ab1b21ee000 ---p 0000d000 fd:00 785832 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libverify.so 2ab1b21ee000-2ab1b21f0000 rw-p 0000d000 fd:00 785832 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libverify.so 2ab1b21f0000-2ab1b221b000 r-xp 00000000 fd:00 785827 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libjava.so 2ab1b221b000-2ab1b241a000 ---p 0002b000 fd:00 785827 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libjava.so 2ab1b241a000-2ab1b241c000 rw-p 0002a000 fd:00 785827 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libjava.so 2ab1b241c000-2ab1b241d000 ---p 00000000 00:00 0 2ab1b241d000-2ab1b241f000 rw-p 00000000 00:00 0 2ab1b241f000-2ab1b2429000 r-xp 00000000 fd:00 785802 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libinstrument.so 2ab1b2429000-2ab1b2628000 ---p 0000a000 fd:00 785802 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libinstrument.so 2ab1b2628000-2ab1b2629000 rw-p 00009000 fd:00 785802 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libinstrument.so 2ab1b2629000-2ab1b2631000 rw-s 00000000 fd:04 393221 /tmp/hsperfdata_wls81/560322 2ab1b2635000-2ab1b2641000 r-xp 00000000 fd:00 654113 /lib64/libnss_files-2.12.so 2ab1b2641000-2ab1b2841000 ---p 0000c000 fd:00 654113 /lib64/libnss_files-2.12.so 2ab1b2841000-2ab1b2842000 r--p 0000c000 fd:00 654113 /lib64/libnss_files-2.12.so 2ab1b2842000-2ab1b2843000 rw-p 0000d000 fd:00 654113 /lib64/libnss_files-2.12.so 2ab1b2843000-2ab1b285d000 r-xp 00000000 fd:00 785824 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libzip.so 2ab1b285d000-2ab1b2a5d000 ---p 0001a000 fd:00 785824 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libzip.so 2ab1b2a5d000-2ab1b2a5e000 rw-p 0001a000 fd:00 785824 /wls/apache/springboot/jdk1.8.0_144/jre/lib/amd64/libzip.so 2ab1b2a5e000-2ab1b2a61000 ---p 00000000 00:00 0 2ab1b2a61000-2ab1b2b5f000 rw-p 00000000 00:00 0 2ab1b2b5f000-2ab1b2b62000 ---p 00000000 00:00 0 2ab1b2b62000-2ab1b3260000 rw-p 00000000 00:00 0 2ab1b3260000-2ab1b3263000 ---p 00000000 00:00 0 2ab1b3263000-2ab1b3361000 rw-p 00000000 00:00 0

GC Activity changes s] 2022-04-13T13:41:03.383+0800: 96.992: [GC (Allocation Failure) 2022-04-13T13:41:03.383+0800: 96.993: [ParNew: 233370K->26176K(235968K), 0.0956726 secs] 629477K->430367K(1022400K), 0.0959928 secs] [Times: user=0.16 sys=0.02, real=0.10 sec s] 2022-04-13T13:41:06.186+0800: 99.795: [GC (Allocation Failure) 2022-04-13T13:41:06.186+0800: 99.795: [ParNew: 235968K->26176K(235968K), 0.0762159 secs] 640159K->438802K(1022400K), 0.0764079 secs] [Times: user=0.14 sys=0.02, real=0.07 sec s] 2022-04-13T13:41:29.728+0800: 123.338: [GC (Allocation Failure) 2022-04-13T13:41:29.728+0800: 123.338: [ParNew: 235968K->17240K(235968K), 0.0759160 secs] 648594K->441899K(1022400K), 0.0761350 secs] [Times: user=0.10 sys=0.01, real=0.07 s ecs] 2022-04-13T13:41:33.781+0800: 127.390: [GC (Allocation Failure) 2022-04-13T13:41:33.781+0800: 127.390: [ParNew: 227032K->14330K(235968K), 0.0313353 secs] 651691K->438990K(1022400K), 0.0315318 secs] [Times: user=0.06 sys=0.00, real=0.03 s ecs] 2022-04-13T13:41:34.257+0800: 127.867: [GC (Allocation Failure) 2022-04-13T13:41:34.258+0800: 127.867: [ParNew: 224122K->19108K(235968K), 0.0231734 secs] 648782K->443767K(1022400K), 0.0233857 secs] [Times: user=0.04 sys=0.00, real=0.03 s ecs] 2022-04-13T13:41:34.968+0800: 128.577: [GC (Allocation Failure) 2022-04-13T13:41:34.968+0800: 128.577: [ParNew: 228900K->16513K(235968K), 0.0513644 secs] 653559K->441173K(1022400K), 0.0515632 secs] [Times: user=0.10 sys=0.00, real=0.06 s ecs] 2022-04-13T13:41:35.391+0800: 129.000: [GC (Allocation Failure) 2022-04-13T13:41:35.391+0800: 129.000: [ParNew: 226305K->16142K(235968K), 0.0329671 secs] 650965K->440802K(1022400K), 0.0331398 secs] [Times: user=0.04 sys=0.00, real=0.03 s ecs] 2022-04-13T13:41:36.212+0800: 129.821: [GC (Allocation Failure) 2022-04-13T13:41:36.212+0800: 129.821: [ParNew: 225934K->25385K(235968K), 0.0286318 secs] 650594K->451254K(1022400K), 0.0288211 secs] [Times: user=0.05 sys=0.01, real=0.03 s ecs] 2022-04-13T13:41:50.869+0800: 144.478: [GC (Allocation Failure) 2022-04-13T13:41:50.869+0800: 144.478: [ParNew: 235177K->12294K(235968K), 0.0490163 secs] 661046K->444869K(1022400K), 0.0492650 secs] [Times: user=0.08 sys=0.01, real=0.05 s ecs] 2022-04-13T13:45:17.263+0800: 350.872: [GC (Allocation Failure) 2022-04-13T13:45:17.263+0800: 350.872: [ParNew: 222086K->16495K(235968K), 0.0516914 secs] 654661K->451717K(1022400K), 0.0519864 secs] [Times: user=0.10 sys=0.01, real=0.05 s ecs] 2022-04-13T13:51:50.728+0800: 744.337: [GC (Allocation Failure) 2022-04-13T13:51:50.728+0800: 744.337: [ParNew: 226287K->11260K(235968K), 0.0522458 secs] 661509K->447078K(1022400K), 0.0526557 secs] [Times: user=0.10 sys=0.00, real=0.05 s ecs] 2022-04-13T13:52:11.202+0800: 764.812: [GC (Allocation Failure) 2022-04-13T13:52:11.203+0800: 764.812: [ParNew: 221052K->17795K(235968K), 0.0374539 secs] 656870K->454463K(1022400K), 0.0377646 secs] [Times: user=0.07 sys=0.00, real=0.04 s ecs] 2022-04-13T13:52:11.682+0800: 765.292: [GC (Allocation Failure) 2022-04-13T13:52:11.683+0800: 765.292: [ParNew: 227587K->16161K(235968K), 0.0329373 secs] 664255K->454037K(1022400K), 0.0331587 secs] [Times: user=0.06 sys=0.00, real=0.04 s ecs] 2022-04-13T13:52:13.328+0800: 766.937: [GC (Allocation Failure) 2022-04-13T13:52:13.328+0800: 766.937: [ParNew: 225953K->18520K(235968K), 0.0346576 secs] 663829K->457373K(1022400K), 0.0350023 secs] [Times: user=0.07 sys=0.00, real=0.03 s ecs] 2022-04-13T13:55:09.327+0800: 942.936: [GC (CMS Initial Mark) [1 CMS-initial-mark: 438853K(786432K)] 518248K(1022400K), 0.0560021 secs] [Times: user=0.11 sys=0.00, real=0.06 secs] 2022-04-13T13:55:09.383+0800: 942.992: [CMS-concurrent-mark-start] 2022-04-13T13:55:09.801+0800: 943.410: [CMS-concurrent-mark: 0.416/0.418 secs] [Times: user=0.42 sys=0.00, real=0.41 secs] 2022-04-13T13:55:09.801+0800: 943.410: [CMS-concurrent-preclean-start] 2022-04-13T13:55:09.811+0800: 943.420: [CMS-concurrent-preclean: 0.009/0.010 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2022-04-13T13:55:09.811+0800: 943.420: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 2022-04-13T13:55:14.930+0800: 948.539: [CMS-concurrent-abortable-preclean: 2.594/5.119 secs] [Times: user=2.61 sys=0.02, real=5.12 secs] 2022-04-13T13:55:14.931+0800: 948.540: [GC (CMS Final Remark) [YG occupancy: 79873 K (235968 K)]2022-04-13T13:55:14.931+0800: 948.540: [Rescan (parallel) , 0.0434664 secs]2022-04-13T13:55:14.975+0800: 948.584: [weak refs processing, 0.00 66264 secs]2022-04-13T13:55:14.981+0800: 948.590: [class unloading

1057105012 avatar Apr 20 '22 11:04 1057105012

This seems like a JVM bug. What version are you running and did you upgrade to the most recent?

raphw avatar Apr 20 '22 11:04 raphw

I am not sure how to solve this problem, which version should I upgrade to, which version will not report errors. Is currently the jdk1.8.0.144

1057105012 avatar Apr 21 '22 06:04 1057105012

Can you try the latest version and also JDK 11/17? I wonder if this is a bug in the JDK that has been fixed.

raphw avatar Apr 21 '22 11:04 raphw

However, this problem will only occur after I bring my own ByteBuddy agent, which usually occurs an hour or more later. This project has an APM Agent and my own agent, both of which are ByteBuddy. A special feature of my agent is that I have staked file, System and JDK code I found a document https://support.oracle.com/knowledge/Middleware/2552944_1.html#FIX But I don't have permission to see it, because I'm not an Oracle user, so you can help me see it，thank you, this problem is too tricky

1057105012 avatar Apr 21 '22 14:04 1057105012

I assume that you found a bug in OpenJDK, but it is difficult to pin down. If you instrument system code, you might change something that is assumed in the JDK native code and create an inconsistency. I am afraid, you would need to trial and error while changing your instrumentations to see when the error stops.

raphw avatar Apr 21 '22 20:04 raphw