amazon-kinesis-client-python

amazon-kinesis-client-python copied to clipboard

I have a Kinesis Stream Reader setup on an EC2 instance with the an instance profile with assume-role permissions to assume a reader-role to a Kinesis Stream in a separate AWS account. I can access the stream perfectly fine if I configure my credentials by using the STS service through the AWS-CLI, retrieve temporary credentials and setup the ~/.aws/credentials file accordingly. But, I'm running into problems if I try to use the STSAssumeRoleSessionCredentialsProvider in kcl.properties to automatically refresh the temporary credentials.

The credentials in kcl.properties is as follows:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account-id>:role/<role-name>|session-name

But I seem to be running into the following issue.

No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

I'm using the Amazon KCL Version : 1.4.4 with the following JARS:

• amazon-kinesis-client-1.7.5.jar
• aws-java-sdk-cloudwatch-1.11.115.jar
• aws-java-sdk-core-1.11.115.jar
• aws-java-sdk-dynamodb-1.11.115.jar
• aws-java-sdk-kinesis-1.11.115.jar
• aws-java-sdk-kms-1.11.115.jar
• aws-java-sdk-s3-1.11.115.jar
• commons-codec-1.9.jar
• commons-lang-2.6.jar
• commons-logging-1.1.3.jar
• guava-18.0.jar
• httpclient-4.5.2.jar
• httpcore-4.4.4.jar
• ion-java-1.0.2.jar
• jackson-annotations-2.6.0.jar
• jackson-core-2.6.6.jar
• jackson-databind-2.6.6.jar
• jackson-dataformat-cbor-2.6.6.jar
• jmespath-java-1.11.115.jar
• joda-time-2.8.1.jar
• protobuf-java-2.6.1.jar

Any help with respect to missing JARs or configuration specifics would be helpful.

Hi - I'm also trying to assume a role and so far have been unable to. I get the same message as you if I use the same format for AWSCredentialsProvider. I'm curious to know where you found the format in the first place? I haven't been able to find any kind of documentation for it.

Edit: Found the source code: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoder.java#L72

The STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar, so you would need to add the aws-java-sdk-sts jar to the class path.

If you don't mind using Apache Maven this is a pom.xml that will create a launch script that will setup the class path for you.

To create the launcher script:

1. Create an empty directory
2. Create a new file in the directory called pom.xml
3. Install Apache Maven if you don't already have it
4. In the directory created in 1, run the command mvn package
5. If everything works you should have a script in target/appassembler/bin/multilang-daemon, and all the jars are in target/appassembler/repo.
6. Copy the directory target/appassembler to wherever you need it.
7. To start your application run the mutlilang-daemon script with the location of your property file.

@pfifer , after following your suggestion and building multilang-daemo I seem still have an issue

./bin/multilang-daemon config/kcl.local.properties
/usr/bin/java -classpath /home/worker/app/etc:/home/worker/app/repo/com/amazonaws/amazon-kinesis-client/1.7.5/amazon-kinesis-client-1.7.5.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-dynamodb/1.11.115/aws-java-sdk-dynamodb-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-s3/1.11.115/aws-java-sdk-s3-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kms/1.11.115/aws-java-sdk-kms-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kinesis/1.11.115/aws-java-sdk-kinesis-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-cloudwatch/1.11.115/aws-java-sdk-cloudwatch-1.11.115.jar:/home/worker/app/repo/com/google/guava/guava/18.0/guava-18.0.jar:/home/worker/app/repo/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar:/home/worker/app/repo/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/worker/app/repo/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/aws-java-sdk-sts-1.11.129.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-core/1.11.129/aws-java-sdk-core-1.11.129.jar:/home/worker/app/repo/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar:/home/worker/app/repo/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar:/home/worker/app/repo/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/worker/app/repo/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-databind/2.6.6/jackson-databind-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-core/2.6.6/jackson-core-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.6/jackson-dataformat-cbor-2.6.6.jar:/home/worker/app/repo/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/home/worker/app/repo/com/amazonaws/jmespath-java/1.11.129/jmespath-java-1.11.129.jar:/home/worker/app/repo/com/amazonaws/amazon-kinesis-multilang-launcher/1.0-SNAPSHOT/amazon-kinesis-multilang-launcher-1.0-SNAPSHOT.jar -Dapp.name=multilang-daemon -Dapp.pid=15086 -Dapp.repo=/home/worker/app/repo -Dapp.home=/home/worker/app -Dbasedir=/home/worker/app com.amazonaws.services.kinesis.multilang.MultiLangDaemon config/kcl.local.properties
No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

output contains extra echo call to dump classpath before calling exec.

@antgus here is an example https://github.com/awslabs/amazon-kinesis-client/blob/master/src/test/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoderTest.java#L63

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111:role/name|session1

but seems does not work

it seems works with AWSCredentialsProvider = InstanceProfileCredentialsProvider

I've just tested it using the line below (with redactions), and was able to access a stream in another account:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session

I've tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I've updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

1. In you application directory create a directory called logback (you can call it something else if you want to).
2. Download this logback.xml file, and save it to logback/logback.xml
3. Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

I have a slightly different issue with this. I have the jars loaded and it's picking up the STS provider, but it can no longer find the env vars containing my long lived creds:

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.prepare(MultiLangDaemonConfig.java:116) at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:107) at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:77) at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:64) at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.main(MultiLangDaemon.java:131)

The DefaultAWSCredentialsProviderChain works in the same config.

Oh, right, I jumped the gun. That's probably the highly obscure exception raised when the AssumeRole api call gets AccessDenied, which happened when I used the CLI with the same roles.

So, once I fixed the trust relationship, all worked well.

Hello @pfifer!

I'm struggling with the same problem as jazzTheJackRabbit and ykhrustalev. I'm using python. What I tried:

1. Your approach adding aws-java-sdk-sts jar to the class path but receive an error: Can't find any credentials provider matching com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111111111111:role/RoleName

2. Specifying AWSCredentialsProvider = ProfileCredentialsProvider with env var AWS_PROFILE=my_profile_name. But received an error: No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

Do I understand it correctly that KCL does not read configuration from ~/.aws/config and it is useless to create profiles with (assume)roles there? This approach works for me with boto3 library on python.

Any help will be highly appreciated.

Found solution myself. For those experiencing the same problem and using python:

1. I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from /target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

2. In the sample.parameters file I wrote:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain

Previously, I didn't specify the session_name, cause boto3 didn't need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

Hello @pfifer!

This thread has been really helpful to setup the Python KCL with STSAssumeRoleSessionCredentialsProvider. However, we enforce MFA to Assume Roles here.

I am jumping in to know how I can configure this in the properties file. Is it even supported?

Thanks, Pierre

Hello @holmesjr . I have met the same error "Unable to load AWS credentials from any provider in the chain" Can I know how did u fix the trust relationship? Thanks

Hello @pfifer ! I have created an IAM role which defines all the permissions of AWS resources I will use in my application. And in my local environment, I created config file under ~/.aws path as following: [default] region = us-west-2 [profile my_profile_name] role_arn = my_role_arn region = us-west-2 source_profile = default

then I export the AWS_PROFILE variable in the following way: export AWS_PROFILE = my_profile_name
so that the app can run according to the IAM role.

As you mentioned above, STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar. So I manually downloaded aws-java-sdk-sts-1.11.300.jar and added it into the same directory where other jars stored. Then I changed AWSCredentialsProvider property into STSAssumeRoleSessionCredentialsProvider in property file. However, I always had this error "Unable to load AWS credentials from any provider in the chain"

I have two questions:

1. Is it possible to run the consumer code locally (in my laptop environment) with IAM role defined in /.aws/config file and what causes that error?
2. There is a PR https://github.com/awslabs/amazon-kinesis-client-python/pull/70 which let setup.py downloads the jar including sts.jar. Why this PR hasn't be merged? I found the corresponding STSAssumeRoleSessionCredentialsProvider class is deprecated https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html . Is this the reason why setup.py won't download sts jar for developers?

The script uses a defined classpath so just adding the jar won't include it. I would recommend following the instructions in my earlier post.

To answer your second question the STS jar is only required for those using the STS credentials. We don't want to increase the required dependencies unless absolutely necessary.

Hello @pfifer, is there a way to pass external_Id when using STSAssumeRoleSessionCredentialsProvider?

Found solution myself. For those experiencing the same problem and using python:

1. I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from /target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

1. In the sample.parameters file I wrote:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain

Previously, I didn't specify the session_name, cause boto3 didn't need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

I have followed the same , however not sure why still getting this below error - any idea please

Caused by: java.lang.IllegalArgumentException: No credential providers specified at com.amazonaws.auth.AWSCredentialsProviderChain.(AWSCredentialsProviderChain.java:59) at software.amazon.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoder.decodeValue(AWSCredentialsProviderPropertyValueDecoder.java:57) at software.amazon.kinesis.multilang.config.MultiLangDaemonConfiguration.lambda$new$0(MultiLangDaemonConfiguration.java:237) at software.amazon.kinesis.multilang.config.BuilderDynaBean.set(BuilderDynaBean.java:258) at software.amazon.kinesis.multilang.config.MultiLangDaemonConfiguration.setAWSCredentialsProvider(MultiLangDaemonConfiguration.java:172) ... 18 more java.lang.reflect.InvocationTargetException

I've just tested it using the line below (with redactions), and was able to access a stream in another account:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session

I've tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I've updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

1. In you application directory create a directory called logback (you can call it something else if you want to).
2. Download this logback.xml file, and save it to logback/logback.xml
3. Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

this worked for me after hours of errors

If this doesnt work, add sts-sdk manually using this command wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.12.428/aws-java-sdk-sts-1.12.428.jar -O "$APP_ROOT"/kcl/aws-sdk-sts.jar

and append java path of the location KCL_COMMAND="$(python "$APP_ROOT"/kcl/helper.py --print_command --java /usr/bin/java --properties "$APP_ROOT"/kcl/consumer.properties --classpath "$APP_ROOT"/kcl/*.jar)"

Works perfectly 💯

If this doesnt work, add sts-sdk manually using this command wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.12.428/aws-java-sdk-sts-1.12.428.jar -O "$APP_ROOT"/kcl/aws-sdk-sts.jar

and append java path of the location KCL_COMMAND="$(python "$APP_ROOT"/kcl/helper.py --print_command --java /usr/bin/java --properties "$APP_ROOT"/kcl/consumer.properties --classpath "$APP_ROOT"/kcl/*.jar)"

Works perfectly 💯 Thank you @alexevan13 , one variant I did was to include this reference in the pom.xml I have