configurable flush threshold and interval (ms)
context
We (SPI) use the telemetry-core SDK for our Kafka telemetry application that scrapes Kafka clusters and sends metrics to New Relic. i believe we've hit a scenario where the code is generating more metrics than the SDK can handle. this causes buffer pressure that manifest as these warnings:
WARN [2025-08-15 18:01:12,788] com.newrelic.telemetry.LimitingScheduler: Refusing to schedule batch of size 251 (would put us over max size 500000, available = 4)
WARN [2025-08-15 18:01:12,788] com.newrelic.telemetry.LimitingScheduler: DATA IS BEING LOST!
which come from here: https://github.com/newrelic/newrelic-telemetry-sdk-java/blob/main/telemetry-core/src/main/java/com/newrelic/telemetry/LimitingScheduler.java#L36-L46
this is not ideal as it regularly drops data and causes gaps in our Kafka charts as shown below.
This PR adds the ability to configure the flush threshold and flush interval (milliseconds) so that we can flush the internal buffer more frequently and avoid the aforementioned warnings.
usage
TelemetryClient telemetryClient = new TelemetryClient(
metricBatchSender,
spanBatchSender,
eventBatchSender,
logBatchSender,
3, // shutdown seconds
true, // use daemon threads
1_000_000, // max buffer size (1M items)
75, // flush threshold (75% buffer usage)
5000 // flush interval (5 seconds)
);
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.