ozone
ozone copied to clipboard
apache
HDDS-10435. Support S3 object tags for existing requests
What changes were proposed in this pull request?
This patch implements the fundamental changes to support object tags. These includes:
- Key protobuf field addition for
OmKeyInfoto support object tags - Support "x-amz-tagging" to set object tags during PutObject, InitiateMultipartUpload and CopyObject
- Support other tagging related metadata (e.g. x-amz-tagging-directive for copy, x-amz-tagging-count for GetObject)
- Extension to client APIs (old APIs remains unchanged and should still be compatible, client does not need to change anything)
Since this patch already covers fundamental changes, the S3 object tagging APIs (e.g. PutObjectTagging, GetObjectTagging, and DeleteObjectTagging) support will be implemented in the follow up ticket (HDDS-10655).
Note: Currently OmDirectoryInfo does not support tags yet. We might need to support this in the future.
See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10435
How was this patch tested?
Unit, integration, and acceptance tests.
@vtutrinov @SaketaChalamchala @tanvipenumudy Could you help take a look when you have time?
Thanks for addressing the comments @ivandika3 . Just one more question, do the robot tests cover both OBJECT_STORE and FILE_SYSTEM_OPTIMIZED buckets?
Thanks for addressing the comments @ivandika3 . Just one more question, do the robot tests cover both OBJECT_STORE and FILE_SYSTEM_OPTIMIZED buckets?
@SaketaChalamchala I don't think so. Currently it seems "commonawslib.robot" creates bucket using S3 protocol which should default to OBJECT_STORE. We might need to add FSO support for the S3 compatibility test in the future.
I parameterized the integration testing (TestOzoneRpcClientAbstract) to test for different bucket layouts to handle it for now.
I briefly reviewed the code, LGTM. @kerneltime Do you have time to take a look.
BTW, @ivandika3 Are you interested in developing User defined object metadata (https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)? It's very similar to this feature.
Thanks @SaketaChalamchala @guohao-rosicky for the reviews.
BTW, @ivandika3 Are you interested in developing User defined object metadata (https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)?
It should have been implemented in https://github.com/apache/ozone/pull/3728 and https://github.com/apache/ozone/pull/6489. Please let me know if I miss anything. Tagging is similar to custom metadata, but the use case and constraints are different.
One requirement is for the client code to move independently of OM version. It is possible that there is a newer S3G client talking to an older OM. In this case how will a new S3G work with an old OM? We can bump up the version between client and OM so that if the client detects an old OM, it will reject the requests with tags.
Based on the code, the old OM should ignore the tag field. @ivandika3 @kerneltime
We can bump up the version between client and OM so that if the client detects an old OM, it will reject the requests with tags.
@kerneltime Updated, I created a new OzoneManagerVersion for object tag support and add logic to reject requests if OM does not support tags. The new OzoneManagerVersion would also be used for client-side OM version validation for PutObjectTagging, DeleteObjectTagging, and GetObjectTagging in HDDS-10655. PTAL thanks.
@guohao-rosicky is correct that the current behavior is the old OM would just ignore the tag fields. It might be better to ignore than to reject for availability reason, but I'm fine with either approach.
Thanks @ivandika3 for the patch, @guohao-rosicky, @kerneltime, @SaketaChalamchala, @vtutrinov for the review.
Thanks @ivandika3 for the patch. BTW, when you submit https://issues.apache.org/jira/browse/HDDS-10655 Can notify me to review it.