BUG Memory leak when use agent

[ShangfengDing]

I use 0.12.0 agent upload JFR data to my backend.

<dependency>
            <groupId>io.pyroscope</groupId>
            <artifactId>agent</artifactId>
            <version>0.12.0</version>
 </dependency>

Upload code:

Config profileConfig = new Config.Builder()
                    .setFormat(Format.JFR)
                    .setLogLevel(Logger.Level.INFO)
                    .setProfilingEvent(EventType.ITIMER)
                    .setProfilingAlloc("0")
                    .setProfilingInterval(Duration.ofMillis(10))
                    .setUploadInterval(Duration.ofSeconds(5))
                    .build();
GalileoProfileExporter exporter = new GalileoProfileExporter(resource, DEFAULT_RETRY);
PyroscopeAgent.start(new PyroscopeAgent.Options.Builder(profileConfig).setExporter(exporter).build());

My exporter:

public class GalileoProfileExporter implements Exporter {
    private static final Duration TIMEOUT = Duration.ofSeconds(10);
    private static final String ENDPOINT = "https://xxxxxxxxxxxxx";

    final Ocp.Resource resource;
    final int ingestMaxTries;
    final Logger logger = new DefaultLogger(Logger.Level.INFO, System.err);
    final OkHttpClient client = new OkHttpClient.Builder()
            .connectTimeout(TIMEOUT)
            .readTimeout(TIMEOUT)
            .callTimeout(TIMEOUT)
            .build();

    public GalileoProfileExporter(Ocp.Resource resource, int ingestMaxTries) {
        this.resource = resource;
        this.ingestMaxTries = ingestMaxTries;
    }

    @Override
    public void export(Snapshot snapshot) {
        final HttpUrl url = HttpUrl.parse(ENDPOINT);
        logger.log(Logger.Level.INFO, "Upload profile. %s %s JFR: %s",
                snapshot.started.toString(), snapshot.ended.toString(), snapshot.data.length);
        byte[] reqBody = constructReqBody(snapshot);
        byte[] compressedData = new byte[0];
        try {
            compressedData = Snappy.compress(reqBody);
        } catch (IOException e) {
            logger.log(Logger.Level.ERROR, "Error compress snapshot.");
        }
        MediaType type = MediaType.parse("application/octet-stream");
        RequestBody jfrBody = RequestBody.create(compressedData, type);
        Request.Builder request = new Request.Builder()
                .post(jfrBody)
                .url(url);
        request.header("language", "java");

        try (Response response = client.newCall(request.build()).execute()) {
            int status = response.code();
            if (status >= 400) {
                ResponseBody body = response.body();
                final String responseBody;
                if (body == null) {
                    responseBody = "";
                } else {
                    responseBody = body.string();
                }
                logger.log(Logger.Level.ERROR, "Error uploading snapshot: %s %s", status, responseBody);
            }
            if (status == 200) {
                logger.log(Logger.Level.INFO, "Success upload jfr");
            }
        } catch (final IOException e) {
            logger.log(Logger.Level.ERROR, "Error uploading snapshot: %s", e.getMessage());
        }
    }

    private byte[] constructReqBody(Snapshot snapshot) {
        Otp.Profile.Builder profile = Otp.Profile.newBuilder()
                .setName("jfr")
                .setType("cpu")
                .setData(ByteString.copyFrom(snapshot.data));
        Otp.ProfilesBatch profilesBatch = Otp.ProfilesBatch.newBuilder()
                .setSequence(1)
                .setStart(snapshot.started.toEpochMilli() / 1000)
                .setEnd(snapshot.ended.toEpochMilli() / 1000)
                .setResource(this.resource)
                .addProfiles(profile)
                .build();
        return profilesBatch.toByteArray();
    }

}

When I opened the agent report, I found that the memory was constantly rising, and I suspected there was a memory leak situation

Is there anyone who has encountered similar problems with me, or is there a problem with where I wrote it? Thanks

[korniltsev]

can you do a hprof dump and check what's causing the mem consumption?

[ShangfengDing]

can you do a hprof dump and check what's causing the mem consumption?

From top, the RES section has exceeded the 1g memory set by the JVM

I open the Native Memory Tracking

jmap result:

[ShangfengDing]

I use the demo which provided by pyrocope also has memory leak

import io.pyroscope.http.Format;
import io.pyroscope.javaagent.PyroscopeAgent;
import io.pyroscope.javaagent.Snapshot;
import io.pyroscope.javaagent.api.Exporter;
import io.pyroscope.javaagent.api.Logger;
import io.pyroscope.javaagent.config.Config;
import io.pyroscope.javaagent.impl.DefaultConfigurationProvider;
import io.pyroscope.labels.Pyroscope;
import io.pyroscope.labels.LabelsSet;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class App {
    public static final int N_THREADS = 8;

    public static void main(String[] args) {
        PyroscopeAgent.start(
            new PyroscopeAgent.Options.Builder(
                new Config.Builder()
                    .setApplicationName("demo.app{qweqwe=asdasd}")
                    .setServerAddress("http://localhost:4040")
                    .setFormat(Format.JFR)
                    .setLogLevel(Logger.Level.DEBUG)
                    .setLabels(mapOf("user", "tolyan"))
                    .build())
//                .setExporter(new MyStdoutExporter())
                .build()
        );
        Pyroscope.setStaticLabels(mapOf("region", "us-east-1"));

        appLogic();
    }

    private static void appLogic() {
        ExecutorService executors = Executors.newFixedThreadPool(N_THREADS);
        for (int i = 0; i < N_THREADS; i++) {
            executors.submit(() -> {
                Pyroscope.LabelsWrapper.run(new LabelsSet("thread_name", Thread.currentThread().getName()), () -> {
                        while (true) {
                            try {
                                fib(32L);
                            } catch (InterruptedException e) {
                                Thread.currentThread().interrupt();
                                break;
                            }
                        }
                    }
                );
            });
        }
    }

    private static Map<String, String> mapOf(String... args) {
        Map<String, String> staticLabels = new HashMap<>();
        for (int i = 0; i < args.length; i += 2) {
            staticLabels.put(args[i], args[i + 1]);
        }
        return staticLabels;
    }

    private static long fib(Long n) throws InterruptedException {
        if (n == 0L) {
            return 0L;
        }
        if (n == 1L) {
            return 1L;
        }
        Thread.sleep(100);
        return fib(n - 1) + fib(n - 2);
    }

    private static class MyStdoutExporter implements Exporter {
        @Override
        public void export(Snapshot snapshot) {
            System.out.printf("Export %d %d%n", snapshot.data.length, snapshot.labels.toByteArray().length);
        }
    }
}

I use the 1g container to run this program, set -Xms256m -Xmx256m -XX:MaxDirectMemorySize=10m

the memory monitor:

[dvovney]

Any updates on this one? I also faced with memory leak after adding Pyrosope agent

[korniltsev]

@dvovney did you check heap dump?

[dvovney]

@korniltsev yes, I could not figure out what actually caused a leak. at some point we've added pyroscope into several services, and exactly after that all of them started leaking