`Kinesis::GetShardIterator` with `AT_TIMESTAMP` does not work when CBOR is disabled

[dannycranmer]

Describe the bug

When you disable CBOR Kinesis::GetShardIterator using AT_TIMESTAMP does not work.

Expected Behavior

Request is accepted and shard iterator is returned

Current Behavior

com.amazonaws.services.kinesis.model.InvalidArgumentException: The timestampInMillis parameter cannot be greater than the currentTimestampInMillis. timestampInMillis: 1622282661795000, currentTimestampInMillis: 1622300462653 (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidArgumentException; Request ID: cabb1ff7-e5d8-589d-9397-46992fda7f59; Proxy: null)

	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2839)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2806)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2795)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetShardIterator(AmazonKinesisClient.java:1399)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:1370)

Steps to Reproduce

I have tested against SDK v1.11.844 and v1.11.1030

public class TestCbor {

    @Test
    public void testSdkWithCborDisabled() {
        System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");

        testGetShardIterator();
    }

    @Test
    public void testSdkWithCbor() {
        System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "false");

        testGetShardIterator();
    }

    private void testGetShardIterator() {
        AmazonKinesis kinesisClient = AmazonKinesisClientBuilder.standard()
                .withRegion("us-east-1")
                .build();

        GetShardIteratorRequest request = new GetShardIteratorRequest()
                .withStreamName("input-stream")
                .withShardIteratorType(ShardIteratorType.AT_TIMESTAMP)
                .withTimestamp(new Date(1622282661795L))
                .withShardId("shard-00000001");

        assertNotNull(kinesisClient.getShardIterator(request));
    }
}

Possible Solution

Seems to work in v2 and might be related to https://github.com/aws/aws-sdk-java-v2/issues/1021

Context

CBOR is not supported when using kinesalite and therefore CBOR is disabled for end to end tests

Your Environment

Bug was discovered in Flink Kinesis Connector issue, however reproduced in isolated project

[debora-ito]

Thank you for reporting the issue @dannycranmer. We are able to reproduce, we are going to investigate the cause.

[yonigibbs]

When working against localstack (or kinesis-mock or kinesalite, both of which localstack can use) I think that using AT_TIMESTAMP isn't working when cbor is enabled either. The reason is that when serialising the value, the millisecond value is used rather than the second value:

generator.writeNumber(date.getTime());

(See here). In the above getTime returns the value in milliseconds.

If you reference software.amazon.kinesis:amazon-kinesis-client:2.4.3 you see similar code in software.amazon.awssdk.protocols.cbor.internal.SdkCborGenerator:

generator.writeNumber(instant.toEpochMilli());

(Can't find this in GitHub anywhere, sorry.) Again in milliseconds.

Confusingly, in the docs for GetShardIteratorRequest for getTimestamp , we see this:

A time stamp is the Unix epoch date with precision in milliseconds.

This can be misread as "the value should be in milliseconds", but this is followed in the docs by

For example, 2016-04-04T19:58:46.480-00:00 or 1459799926.480.

As can be seen here, the value is actually in seconds, not milliseconds.

In localstack (i.e. kinesalite or mock-kinesis) this leads to an error such as this: ValueError: year 54865 is out of range.

Unfortunately this seems to me to render the Flink Kinesis source as unusable with localstack, as discussed here, at least when trying to use AT_TIMESTAMP as the STREAM_INITIAL_POSITION. It does however work when using TRIM_HORIZON, as that request doesn't involve sending a timestamp to Kinesis when calling getShardIterator.

Also from what I can see this is not fixed in v2 of the SDK.

[HeartSaVioR]

I was going to file the same one (Not about CBOR disabled, but the above comment, CBOR enabled). Please do note that it is not just a confusion on documentation. The spec of CBOR clearly says the numeric value should be <seconds>.<milliseconds>, not <milliseconds>, hence this is a bug on the code which does not respect the spec. The bug exists the same in SDK v2 which I filed the ticket there.

The fix is obvious and I'm happy to submit a PR if we are happy to fix this. I would like to know whether fixing this would affect Kinesis (server side) though.

cc. @debora-ito Wanted to hear the feedback whether we want to fix the bug or not, and if we want to fix that, am I encouraged to submit a PR.

[HeartSaVioR]

Or does AWS use a custom specification of CBOR? If then it's unfortunate, and it'd be nice if there is a parser being published for the custom specification. (python would be nice to integrate with LocalStack.)

[regfaker]

Isn't it this marshaller part that has bad timestampFormat in binding? I've tried it with unixTimestamp and it worked (it was sending "Timestamp":1709026357.673). I don't know where is the source/def for this generated code but I can look ...