hive
hive copied to clipboard
HIVE-28272: Support setting per-session S3 credentials in Warehouse
What changes were proposed in this pull request?
At present, Warehouse
use a fixed conf
and we can not change it in the session connection. But acutully we have supported set session-level conf in HMSHandler, see HMSHandler::setMetaConf
.
Some other info can be found in this previous ticket HIVE-16913.
This PR tries to support set seesion-level conf in Warehouse
, and then we can unblock the usefull case like per-session S3 credentials.
- HMS use
Warehouse
to delete/create hdfs dir, so we should pass s3 credentials toWarehouse
conf. Adding threadlocal conf inWarehouse
so that we can pass session-level conf toWarehouse
synchronously. e.g.set metaconf:fs.s3a.secret.key=my_secret_key;
- Make S3 related credentials param valid in
MetastoreConf.ConfVars
Why are the changes needed?
Currently, users can only set S3 credentials on the HMS server side and users cannot set different S3 credentials for multiple S3 buckets. With this change, users can set per-session S3 credentials in client side. e.g, we can set S3 credentials like this from beeline client:
set fs.s3a.secret.key=my_secret_key;
set fs.s3a.access.key=my_access.key;
set fs.s3a.endpoint=my_endpoint;
set metaconf:fs.s3a.secret.key=my_secret_key;
set metaconf:fs.s3a.access.key=my_access_key;
set metaconf:fs.s3a.endpoint=my_endpoint;
And you can try to create/insert a s3a table:
create table testtbls3(id int) location 's3a://bucket/testdb/testtbls3';
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
Local test.