newrelic-telemetry-sdk-java icon indicating copy to clipboard operation
newrelic-telemetry-sdk-java copied to clipboard

configurable flush threshold and interval (ms)

Open nr-phil opened this issue 4 months ago • 2 comments

context

We (SPI) use the telemetry-core SDK for our Kafka telemetry application that scrapes Kafka clusters and sends metrics to New Relic. i believe we've hit a scenario where the code is generating more metrics than the SDK can handle. this causes buffer pressure that manifest as these warnings:

WARN [2025-08-15 18:01:12,788] com.newrelic.telemetry.LimitingScheduler: Refusing to schedule batch of size 251 (would put us over max size 500000, available = 4)

WARN [2025-08-15 18:01:12,788] com.newrelic.telemetry.LimitingScheduler: DATA IS BEING LOST!

which come from here: https://github.com/newrelic/newrelic-telemetry-sdk-java/blob/main/telemetry-core/src/main/java/com/newrelic/telemetry/LimitingScheduler.java#L36-L46

this is not ideal as it regularly drops data and causes gaps in our Kafka charts as shown below.

KPT-Kafka-Cluster-us-ridged-chips-kafka-New-Relic-08-15-2025_11_52_AM

This PR adds the ability to configure the flush threshold and flush interval (milliseconds) so that we can flush the internal buffer more frequently and avoid the aforementioned warnings.


usage

TelemetryClient telemetryClient = new TelemetryClient(
    metricBatchSender,
    spanBatchSender, 
    eventBatchSender,
    logBatchSender,
    3,           // shutdown seconds
    true,        // use daemon threads
    1_000_000,   // max buffer size (1M items)
    75,          // flush threshold (75% buffer usage)
    5000         // flush interval (5 seconds)
);

nr-phil avatar Aug 15 '25 18:08 nr-phil

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 15 '25 18:08 CLAassistant

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Aug 15 '25 18:08 CLAassistant