Bytebuddy hook Infinite loop causes metaspace overflow

[1057105012]

When I use transmittable-thread-local, pinpoint, and my metric agent mount it to the Apollo project, I get the metaspace overflow, I wonder how to fix it, I used the Transform provided by ByteBuddy, based on the interface hook . These are all open source projects on Github, I used the interface to hook SQL startup argument

if [ "$1" = "start" ] ; then echo "==== starting service ====" echo "Service logging file is $SERVICE_LOG" export APP_NAME="apollo-service" export ALI_OPTS="-javaagent:/root/code/apollo-build-scripts/transmittable-thread-local-2.13.0-SNAPSHOT.jar" export METRIC_OPTS="-javaagent:/root/code/apollo-build-scripts/agent.jar export VERSION="2.4.0" export AGENT_PATH="/root/code/apollo-build-scripts/pinpoint-agent-$VERSION" export PINPOINT_OPTS="-javaagent:$AGENT_PATH/pinpoint-bootstrap-$VERSION.jar -Dpinpoint.agentId=apollo-agent -Dpinpoint.applicationName=apollo -Dpinpoint.container=apollo -Dpinpoint.profiler.profiles.active=release" echo $RASP_OPTS echo $PINPOINT_OPTS echo $ALI_OPTS export JAVA_OPTS="$ALI_OPS $METRIC_OPTS $PINPOINT_OPTS $SERVER_JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=30000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms1024M -Xmx1024M -Xmn256M -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=350m -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc -Xloggc:/root/code/apollo-build-scripts/xmrasp/logs/gc.log -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/root/code/apollo-build-scripts/xmrasp/logs/herr.log -Djava.net.preferIPv4Stack=true -Dpafa.log.home=/root/code/apollo-build-scripts/xmrasp/logs -Dlogging.file.name=./apollo-service.log -Dspring.datasource.url=$apollo_config_db_url -Dspring.datasource.username=$apollo_config_db_username -Dspring.datasource.password=$apollo_config_db_password"

log info [2022-06-11 15:05:21.976] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.976] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.976] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.976] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.984] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.985] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.985] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.985] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.993] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.993] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.993] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:21.994] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.002] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.002] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.002] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.002] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.011] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.012] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.012] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.012] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.020] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.021] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.021] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.021] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.029] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.029] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.029] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.029] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.037] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.038] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.038] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.038] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.046] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.046] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.047] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.047] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.055] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.055] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.055] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.055] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.063] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.064] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.064] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.064] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.072] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.072] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.073] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.073] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.081] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.081] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.081] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.081] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.089] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.089] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.090] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.090] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.108] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.108] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.108] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.109] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.120] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.120] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.120] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.121] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.130] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.131] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.131] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.131] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.139] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.139] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.140] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.140] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.148] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.148] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.149] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.149] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.157] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.157] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.158] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.158] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.166] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.166] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.167] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.167] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.177] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.178] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.178] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeUpdate com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.178] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement addBatch com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.189] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement execute com.mysql.cj.jdbc.ServerPreparedStatement [2022-06-11 15:05:22.189] [2040154:metric-logger_deamon_task] DEBUG cn.ope.metric.instrumentation.Transformer: hook info java.sql.Statement executeQuery com.mysql.cj.jdbc.ServerPreparedStatement

jvminfo

A fatal error has been detected by the Java Runtime Environment:

Internal Error (debug.cpp:308), pid=2029860, tid=0x00007f88b4c14700

fatal error: OutOfMemory encountered: Metaspace

JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x00007f89109bc800): JavaThread "metric-timer_task" daemon [_thread_in_vm, id=2029982, stack(0x00007f88b4b14000,0x00007f88b4c15000)]

Stack: [0x00007f88b4b14000,0x00007f88b4c15000], sp=0x00007f88b4c11730, free space=1013k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xacb18a] VMError::report_and_die()+0x2ba V [libjvm.so+0x4ff680] report_java_out_of_memory(char const*)+0x150 V [libjvm.so+0x8a443c] Metaspace::report_metadata_oome(ClassLoaderData*, unsigned long, MetaspaceObj::Type, Metaspace::MetadataType, Thread*)+0x5c V [libjvm.so+0x8ab3f7] Metaspace::allocate(ClassLoaderData*, unsigned long, bool, MetaspaceObj::Type, Thread*)+0x2d7 V [libjvm.so+0x4f32a6] ConstMethod::allocate(ClassLoaderData*, int, InlineTableSizes*, ConstMethod::MethodType, Thread*)+0xf6 V [libjvm.so+0x8b4288] Method::allocate(ClassLoaderData*, int, AccessFlags, InlineTableSizes*, ConstMethod::MethodType, Thread*)+0x38 V [libjvm.so+0x4621bf] ClassFileParser::parse_method(bool, AccessFlags*, Thread*)+0x173f V [libjvm.so+0x462953] ClassFileParser::parse_methods(bool, AccessFlags*, bool*, bool*, Thread*)+0x293 V [libjvm.so+0x4637da] ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>, TempNewSymbol&, bool, Thread)+0xcca V [libjvm.so+0xa3ef0a] SystemDictionary::parse_stream(Symbol*, Handle, Handle, ClassFileStream*, KlassHandle, GrowableArray<Handle>, Thread)+0xda V [libjvm.so+0x7bf919] VM_RedefineClasses::load_new_class_versions(Thread*)+0x389 V [libjvm.so+0x7c0490] VM_RedefineClasses::doit_prologue()+0xc0 V [libjvm.so+0xad27cb] VMThread::execute(VM_Operation*)+0x16b V [libjvm.so+0x78d573] JvmtiEnv::RetransformClasses(int, _jclass* const*)+0x2d3 V [libjvm.so+0x74ef1d] jvmti_RetransformClasses+0x14d C [libinstrument.so+0x4020] retransformClasses+0x1e0 J 17394 sun.instrument.InstrumentationImpl.retransformClasses0(J[Ljava/lang/Class;)V (0 bytes) @ 0x00007f88f462f63e [0x00007f88f462f580+0xbe] J 17311 C1 sun.reflect.GeneratedMethodAccessor220.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (68 bytes) @ 0x00007f88f4c1626c [0x00007f88f4c15dc0+0x4ac] J 9398 C2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007f88f5c1024c [0x00007f88f5c101a0+0xac] J 17393 C1 cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(Ljava/lang/instrument/Instrumentation;[Ljava/lang/Class;)V (66 bytes) @ 0x00007f88f55a5c8c [0x00007f88f55a59e0+0x2ac] J 17392 C1 cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Ljava/util/List;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;)V (60 bytes) @ 0x00007f88f544f114 [0x00007f88f544e9c0+0x754] j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$BatchAllocator;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$Listener;I)I+67 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$PoolStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$DiscoveryStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$BatchAllocator;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LambdaInstrumentationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$DescriptionStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$FallbackStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RawMatcher;)V+451 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RawMatcher;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+179 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(Ljava/lang/instrument/Instrumentation;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+40 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(Ljava/lang/instrument/Instrumentation;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+5 j cn.ope.metric.session.Session.lambda$maybeShouldInit$2()V+18 j cn.ope.metric.session.Session$$Lambda$1552.run()V+4 J 14578 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007f88f621fa64 [0x00007f88f621f9e0+0x84] J 14928 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007f88f528f4cc [0x00007f88f528f280+0x24c] J 14901 C1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (59 bytes) @ 0x00007f88f6af8be4 [0x00007f88f6af85a0+0x644] j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056 V [libjvm.so+0x6943a1] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321 V [libjvm.so+0x694847] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47 V [libjvm.so+0x72fbd0] thread_entry(JavaThread*, Thread*)+0xa0 V [libjvm.so+0xa79753] JavaThread::thread_main_inner()+0x103 V [libjvm.so+0xa7989c] JavaThread::run()+0x11c V [libjvm.so+0x92a338] java_start(Thread*)+0x108 C [libpthread.so.0+0x817a] start_thread+0xea

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 17394 sun.instrument.InstrumentationImpl.retransformClasses0(J[Ljava/lang/Class;)V (0 bytes) @ 0x00007f88f462f5cc [0x00007f88f462f580+0x4c] J 17311 C1 sun.reflect.GeneratedMethodAccessor220.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (68 bytes) @ 0x00007f88f4c1626c [0x00007f88f4c15dc0+0x4ac] J 9398 C2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007f88f5c1024c [0x00007f88f5c101a0+0xac] J 17393 C1 cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(Ljava/lang/instrument/Instrumentation;[Ljava/lang/Class;)V (66 bytes) @ 0x00007f88f55a5c8c [0x00007f88f55a59e0+0x2ac] J 17392 C1 cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Ljava/util/List;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;)V (60 bytes) @ 0x00007f88f544f114 [0x00007f88f544e9c0+0x754] j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$BatchAllocator;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$Listener;I)I+67 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$CircularityLock;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$PoolStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LocationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$DiscoveryStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$BatchAllocator;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RedefinitionStrategy$Listener;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$LambdaInstrumentationStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$DescriptionStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$FallbackStrategy;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RawMatcher;)V+451 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(Ljava/lang/instrument/Instrumentation;Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/AgentBuilder$RawMatcher;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+179 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(Ljava/lang/instrument/Instrumentation;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+40 j cn.ope.metric.thirdparty.net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(Ljava/lang/instrument/Instrumentation;)Lcn/ope/metric/thirdparty/net/bytebuddy/agent/builder/ResettableClassFileTransformer;+5 j cn.ope.metric.session.Session.lambda$maybeShouldInit$2()V+18 j cn.ope.metric.session.Session$$Lambda$1552.run()V+4 J 14578 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007f88f621fa64 [0x00007f88f621f9e0+0x84] J 14928 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007f88f528f4cc [0x00007f88f528f280+0x24c] J 14901 C1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (59 bytes) @ 0x00007f88f6af8be4 [0x00007f88f6af85a0+0x644] j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

[raphw]

You could try to break your transformation down to smaller chuncks by registering a batch strategy. It might reduce presure on metaspace.

[vinayakbmalagatti]

You could try to break your transformation down to smaller chuncks by registering a batch strategy. It might reduce presure on metaspace.

Can you please explain more on this and if possible please share the docs or any other article which explains with the example.

[raphw]

If you register a retransformation strategy, in the next steps you can add refinements. One step allows you to apply a batch.