Configuration to disable processing_statistics_logger

[x418]

We have been using KPL/KCL to produce and consume messages via Kinesis Stream and are quite impressed and happy with it. The one thing that's been rather annoying is the processing_statistics_logger that logs stats every 15 seconds or so. It barely adds any value (we use Cloudwatch metrics) and frankly, ends up polluting the logs.

2017-09-20 13:42:30,054 [kpl-daemon-0006] [trid=, url=] logLevel=INFO  com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2017-09-20 13:42:30.054006] [0x00002547][0x00007f88a0ffb700] [info] [processing_statistics_logger.cc:109] Stage 1 Triggers: { stream: 'stream-name-removed', manual: 0, count: 0, size: 0, matches: 0, timed: 0, UserRecords: 0, KinesisRecords: 0 }
.................................
2017-09-20 13:42:45,054 [kpl-daemon-0006] [trid=, url=] logLevel=INFO  com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2017-09-20 13:42:45.054246] [0x00002547][0x00007f88a0ffb700] [info] [processing_statistics_logger.cc:109] Stage 1 Triggers: { stream: 'stream-name-removed', manual: 0, count: 0, size: 0, matches: 0, timed: 0, UserRecords: 0, KinesisRecords: 0 }

Would it be possible to have it configurable when creating the KinesisProducerConfiguration?

We can obviously set the Log level of com.amazonaws.services.kinesis.producer.LogInputStreamReader to ERROR in our log4j/logback config to remove it (which would remove it altogether). But, we would prefer for it to be configurable.

Also, setting the LogLevel property for kinesis producer configuration did not have any effect on the statistics logger.

Relevant versions: amazon-kinesis-producer: 0.12.5 amazon-kinesis-client: 1.8.1

[bowenli86]

@pfifer can you suggest a graceful way to disable KPL logs? As said above, setting the LogLevel property for kinesis producer configuration did not have any effect on the statistics logger.

[pfifer]

For the short term you can set the logging level of for LogInputStreamReader to WARN or higher.

I do intend to add the ability to configure the reporting from the processing_statistics, though it is still being prioritized.

[lc-nyovchev]

Would it be a programmatic thing? It'd be very nice if we can configure the verbosity of the logger, cause mostly we are interested in warn and error ones, but not so much in info and below.

[idaamit]

@pfifer Hi, can you please guide how to remove the INFO logs in kpl & kcl?

[ObviousDWest]

While you are thinking about metrics logging...

Our configuration does not have cloudWatch uploading enabled. So KPL spams the logs pretty heavily in that case. It would be nice to disable the upload request so the error isn't reported. And it would be nice to see the metrics logged locally, so I don't want to totally disable them.

E.g. 2018-05-17 17:01:28.923 WARN 12147 --- [kpl-daemon-0006] c.a.s.k.producer.LogInputStreamReader : [2018-05-17 17:01:28.923519] [0x00002fc5][0x00007fd5fe7fc700] [warning] AWS Log: WARNEncountered AWSError AccessDenied User: arn:aws:iam::xxx:user/uuu is not authorized to perform: cloudwatch:PutMetricData:

2018-05-17 17:01:28.955 WARN 12147 --- [kpl-daemon-0006] c.a.s.k.producer.LogInputStreamReader : [2018-05-17 17:01:28.955283] [0x00002fc5][0x00007fd5ff7fe700] [warning] AWS Log: WARNEncountered AWSError AccessDenied User: arn:aws:iam::xxx:user/uuu is not authorized to perform: cloudwatch:PutMetricData:

2018-05-17 17:01:29.402 WARN 12147 --- [kpl-daemon-0006] c.a.s.k.producer.LogInputStreamReader : [2018-05-17 17:01:29.401994] [0x00002fc5][0x00007fd5ff7fe700] [warning] AWS Log: WARNEncountered AWSError AccessDenied User: arn:aws:iam::xxx:user/uuu is not authorized to perform: cloudwatch:PutMetricData:

[pfifer]

@ObviousDWest Do you metrics disabled in the KPL configuration?

[ObviousDWest]

I found: MetricsLevel=none in KinesisProducerConfiguration. That seems to have managed the spammy warnings about CloudWatch, while still leaving me the local processing_statistics_logger.cc statistics reported in the logs.

Refer to other config settings here: https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties

[maheshsirigi]

I found: MetricsLevel=none in KinesisProducerConfiguration. That seems to have managed the spammy warnings about CloudWatch, while still leaving me the local processing_statistics_logger.cc statistics reported in the logs.

Refer to other config settings here: https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties

logs are still throttling after setting metrics level to none. Did you do any work around to resolve this issue?

[af-inet]

I do intend to add the ability to configure the reporting from the processing_statistics, though it is still being prioritized.

Maybe processing_statistics could be made to respect the log level thats already being passed to this daemon?

[kamaldon]

Is there any update to this? Setting the metricsLevel to "none" and logLevel to "error" in the KinesisProducerConfiguration still shows the processing_statistics_logger logs which don't provide much value

[yndai]

I've opened https://github.com/awslabs/amazon-kinesis-producer/pull/288 to properly pass on the log level from the main app to the daemon process. Open to any suggestions.

[Magnus1987]

We were able to solve this by setting the log level from KinesisProducerConfiguration().setLogLevel("error"). The issue for us I believe is that we configured log4j to use the format "[%5-level] ..." which will print out [ERROR] but it can also leave trailing whitespaces such as [INFO ] and [WARN ].

The regex pattern in LogInputStreamReader is case insensitive but it is sensitive to any trailing whitespaces within your brackets of the logged level. That means [INFO ] cannot be matched but [ERROR] can be caught.

private static final Pattern LEVEL_REGEX = Pattern.compile(
            "\\[(?<level>trace|debug|info|warn(?:ing)?|error|fatal)\\]", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE);