Persist State Across App Shutdowns/Credentials Store

[BradRuderman]

2 Questions:

1. How can I persist the last processed sequence id, should I just store and initialize a KCL with that sequence id, as part of the initialize and shutdown methods? I am looking to be able to shutdown the KCL app and then restart it and it pick up where it left off. I know while the app is running it persists in DynamoDb but would be great to also persist across shutdowns?

2. Is it possible to programatically get the credentials. We use a special secret vault, I would love a way to pass credentials in during the initialize or startup method versus having to hard code them into the environment. Another option I thought about is to wrap the KCL library and programatically add them to the environment before calling kcl.KCLProcess()

[pfifer]

1. The KCL automatically tracks the last checkpointed sequence number. This is stored in a DynamoDB table, and used to restart processing from that position. The Java documentation for processRecords talks about checkpointing.
2. You can provide a custom credentials provider, but there are some caveats:

   1. The credential provider must be implemented in Java and implement the com.amazonaws.auth.AWSCredentialsProvider interface.

   2. The credential provider needs to be added to the classpath

   3. In your properties file you would need to specify the full class name of the provider e.g. AWSCredentialsProvider=com.amazonaws.auth.AWSCredentialsProviderChain

      See the KinesisClientLibConfigurator for how this is implemented