Noah Potash

I started taking a look at this myself, and ended up down the rabbit hole. Upgrading the aws sdk reference for operator-metering itself was straightforward, and I started things up...

Thanks for clarifying which services need to be targeted. For the java1 sdk, 1.11.623 is the minimum to support the new authentication mechanism. I started by making a simple docker...

Ok, I've put the new jar there (as well as in /opt/hive/lib just to be sure, since the symlink gets broken when I remove the file, I guess) and confirmed...

I figured out what's going on here. First, I rebuilt the ansible-operator image using a modified hive-configmap.yaml that enabled the debug logging for s3a. This confirmed that s3a was using...

DefaultAWSCredentialsProviderChain does indeed fall back to WebIdentityTokenCredentialsProvider, but Hadoop/s3a doesn't use DefaultAWSCredentialsProviderChain . It uses its own ( https://github.com/operator-framework/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java ) and populates it here: https://github.com/operator-framework/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L558 I'm working on rebuilding with...

Cool. Yeah that ought to work, though since we would still need to rebuild Hadoop to pick up the new SDK it probably makes sense to put it in the...

Right - patching over the docker image like I've been doing.

I managed to get everything working! Here's what I did: * Hadoop ○ Rebuilt with the latest SDK and added credentials.add(WebIdentityTokenCredentialsProvider.create()); to S3AUtils.java (should be avoidable when aws/aws-sdk-java#2121 is accepted...

Could you show me an example for how to add an annotation at "deployment" time? The only avenue for customization I saw was metering-custom.yaml, and it wasn't obvious to me...

The service account is bound to the IAM role via annotation, and it is from there that the token gets projected into the pod. The full technical details are in...