secor
secor copied to clipboard
pinterest
Add glue metastore update
This PR adds the possibility to integrate with the AWS Glue metastore also to a fine tune AWS authentification.
Are you planning to merge this PR to upstream? Looks like it's quite a big change which might need some explanation and discussion.
Hi, yes I would like to contribute it. The changes include 2 features :
- A GluePartitionFinalizer (~managed hive metastore on AWS). To reuse code and keep the same logic of the QuboleFinalizer we abstracted it to AbstractPartitionFinalizer leaving the addPartition method to be implemented by the impl without breaking the default behaviour.
- The Glue client required providing a custom credentialsProvider (AssumeRoleAndDefaultCredentialsProviderChain) to be able to do an assume role without breaking the authentication of the S3Uploader. So we added the ability to override the Credentials Provider with "fs.s3a.aws.credentials.provider" and as AWS regions and enpoints are interchangeable we added the abylity to provide one or the other.
No but I can pick it up if there is renewed interest in this.
Le lun. 25 mai 2020 à 21:54, Henry Haiying Cai [email protected] a écrit :
@Sab8 https://github.com/Sab8 Are you still working on this PR?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/pull/921#issuecomment-633699259, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHI4SPHNRAQ5GFZWSZERRDRTLEGTANCNFSM4IUI6TOA .
I think it would be nice to have this PR merged if you already used in your production environment. I think you just need to resolve the conflicts with the current secor code base.