amazon-kinesis-client-python icon indicating copy to clipboard operation
amazon-kinesis-client-python copied to clipboard

Follow up question: how to use initialize_input parameter

Open jiawen-earnin opened this issue 7 years ago • 1 comments

@pfifer Thanks for your reply in this issue https://github.com/awslabs/amazon-kinesis-client-python/issues/81 . The document of AWS kinesis said that 'The KCL calls the initialize method when the record processor is instantiated, passing a specific shard ID as a parameter. This record processor processes only this shard' (https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-implementation-app-py.html). So if I develop my application in python, how can I scale it when there are large volume of data ?

jiawen-earnin avatar Nov 13 '18 18:11 jiawen-earnin

The KCL will pass in the shard that will be processed. The Java process that controls all of this uses threads and will start a new process for each acquired shard.

pfifer avatar Nov 13 '18 20:11 pfifer