Kinesis: PutRecords API limits not enforced

[rob-fin]

I recently noticed that Moto does not enforce the API limits for PutRecords in Kinesis Data Streams. Relevant documentation.

Examples:

Record data exceeds 1 MB

client = boto3.client('kinesis')
client.create_stream(StreamName='my_stream', ShardCount=1)
resp = client.put_records(
    Records=[
        {
            'Data': b'a' * (2**20 + 1),
            'PartitionKey': 'key'
        }
    ],
    StreamName='my_stream'
)
print(resp)

Expected (AWS):

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutRecords operation: 1 validation error detected: Value at 'records.1.member.data' failed to satisfy constraint: Member must have length less than or equal to 1048576

Actual (Moto):

{'FailedRecordCount': 0, 'Records': [{'SequenceNumber': '1', 'ShardId': 'shardId-000000000000'}], 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}

Records exceed 5 MB (including partition keys)

client = boto3.client('kinesis')
client.create_stream(StreamName='my_stream', ShardCount=1)
resp = client.put_records(
    Records=[
        {
            'Data': b'a' * 2**20,
            'PartitionKey': 'key'
        }
    ] * 5,
    StreamName='my_stream'
)
print(resp)

Expected (AWS):

botocore.errorfactory.InvalidArgumentException: An error occurred (InvalidArgumentException) when calling the PutRecords operation: Records size exceeds 5 MB limit

Actual (Moto):

{'FailedRecordCount': 0, 'Records': [{'SequenceNumber': '1', 'ShardId': 'shardId-000000000000'}, {'SequenceNumber': '2', 'ShardId': 'shardId-000000000000'}, {'SequenceNumber': '3', 'ShardId': 'shardId-000000000000'}, {'SequenceNumber': '4', 'ShardId': 'shardId-000000000000'}, {'SequenceNumber': '5', 'ShardId': 'shardId-000000000000'}], 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}

Too many records

client = boto3.client('kinesis')
client.create_stream(StreamName='my_stream', ShardCount=1)
resp = client.put_records(
    Records=[
        {
            'Data': b'a',
            'PartitionKey': 'key'
        }
    ] * 501,
    StreamName='my_stream'
)
print(resp)

Expected (AWS):

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutRecords operation: 1 validation error detected: Value '[PutRecordsRequestEntry(data=java.nio.HeapByteBuffer[pos=0 lim=1 cap=1], explicitHashKey=null, partitionKey=key), ... , PutRecordsRequestEntry(data=java.nio.HeapByteBuffer[pos=0 lim=1 cap=1], explicitHashKey=null, partitionKey=key)]' at 'records' failed to satisfy constraint: Member must have length less than or equal to 500

Actual (Moto):

{'FailedRecordCount': 0, 'Records': [{'SequenceNumber': '1', 'ShardId': 'shardId-000000000000'}, ... , {'SequenceNumber': '501', 'ShardId': 'shardId-000000000000'}], 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}

I'm willing to have a go at it myself.

I notice that PutRecords is not actually documented to raise ValidationExceptions, and the exception class for the Kinesis mock seems to be a bit hard-coded to handle error messages for SplitShard, which is documented to raise it. But my impression for this project is that observed AWS behavior takes precedence over documented. Is this correct?

[bblommers]

Hi @rob-fin, thanks for raising this, and welcome to Moto! PR's are always welcome. There is some documentation on how to get started here: http://docs.getmoto.org/en/latest/docs/contributing/index.html - happy to help out if you need any other info.

observed AWS behavior takes precedence over documented. Is this correct?

That is correct. The AWS documentation tends to be WIP across all services, so if there's anything unclear we try to mimick the actual behaviour.