KPL Internally reported errors

[bisoldi]

I am running the KPL (0.12.9) on C5.XL and M5.L EC2 instances running in us-east-1, processing logfiles in response to SQS messages (the difference between the different size instances being the size of the log files being processed). Overall, I'm finding near perfect performance, except for a handful of exceptions being thrown (and I believe swallowed) by the KPL.

My code:

protected final Semaphore backPressure = new Semaphore(50000, true);

protected void writeRecord(ObjectNode node) throws JsonProcessingException {
	final ByteBuffer byteBuffer = ByteBuffer.wrap(objectMapper.writeValueAsBytes(node));
	
	try {
		backPressure.acquire();
		
		ListenableFuture<UserRecordResult> future = kinesisProducer.addUserRecord(appConfig.streamName, 
                    UUID.randomUUID().toString(), byteBuffer);
		
		Futures.addCallback(future, new FutureCallback<UserRecordResult>() {
			@Override
			public void onFailure(Throwable t) {
                                    // Emit Exception Metrics					
				backPressure.release();
			}

			@Override
			public void onSuccess(UserRecordResult result) {
				// do nothing
				backPressure.release();
			}
		});
	} catch (InterruptedException e) {
		log.error(e.getLocalizedMessage(), e);
	}
}

My configuration (for both size instances):

KinesisProducerConfiguration config = new KinesisProducerConfiguration()
  .setRecordMaxBufferedTime(25000)
  .setCredentialsProvider(credentials)
      .setMaxConnections(24)
      .setRecordTtl(3600000)
      .setRequestTimeout(60000)
      .setRegion("us-east-1");

The errors I'm seeing are:

2018-11-02T10:38:36 [ERROR] com.amazonaws.services.kinesis.producer.LogInputStreamReader :: [2018-11-02 10:38:36.015728] [0x00000622][0x00007f3046ffd700] [error] AWS Log: ERRORCurl returned error code 28

2018-11-02T10:28:49 [WARN] com.amazonaws.services.kinesis.producer.LogInputStreamReader :: [2018-11-02 10:28:49.668323] [0x00005477][0x00007fa6ce7fc700] [warning] AWS Log: WARNEncountered Unknown AWSError Unable to generate a proper httpResponse from the response stream. Response code: 500:

2018-10-31T23:19:32 [WARN] com.amazonaws.services.kinesis.producer.LogInputStreamReader :: [2018-10-31 23:19:32.017578] [0x00001a97][0x00007fdc8cff9700] [warning] AWS Log: WARNEncountered AWSError ServiceUnavailableException:

None of these exceptions appear to be bubbling up to my FutureCallback, so is it safe to assume that the KPL successfully retried the records that originally threw those exceptions?

In the last 2 days, I've only logged a handful of those exceptions; is there anything I should be doing differently to avoid these exceptions completely or is the KPL handling them and I should just continue monitoring for failed records?

[lc-nyovchev]

Having the same problem. Any movement on the issue?

[chris-okorodudu]

I'm having a similar problem. Were you able to determine if the records were indeed retried successfully or find out how to avoid these exceptions?