HDDS-7113. Support overriding response header values

[xichen01]

What changes were proposed in this pull request?

Implement "override GET response header" feature of Ozone S3. Based on this feature, you can override values for a set of response headers using the following query parameters. For example, for a video URL, you can control the behavior of the URL through this feature to play the video or download the video.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7113

How was this patch tested?

This example overrides the values of Content-Type and Content-Disposition in the response header by adding query parameters 'response-content-disposition=inline' and 'response-content-type=video'. This example generates a URL that will play the video 'video.mp4' directly in the browser This is one application of this feature

import boto3


# If authentication is not enabled, aws_access_key_idand aws_secret_access_keycan be ignored
s3 = boto3.client("s3", aws_access_key_id='your_key', aws_secret_access_key='your_secret',
                  endpoint_url='https://xxxxx', region_name='dummyregion')

# Pass in query parameters here
# Upload the video "video.mp4" to volume: "s3v", bucket: "test-bucket"
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'test-bucket', 'Key': 'video.mp4', 'ResponseContentDisposition': 'inline',
            'ResponseContentType': 'video/mp4'},
    ExpiresIn=3600
)


'''
Support Parameters:
    ResponseCacheControl (string) -- Sets the Cache-Control header of the response.
    ResponseContentDisposition (string) -- Sets the Content-Disposition header of the response
    ResponseContentEncoding (string) -- Sets the Content-Encoding header of the response.
    ResponseContentLanguage (string) -- Sets the Content-Language header of the response.
    ResponseContentType (string) -- Sets the Content-Type header of the response.
    ResponseExpires (datetime) -- Sets the Expires header of the response.
'''

print(url)
# https:/xxx/test-bucket/video.mp4?response-content-disposition=inline&response-content-type=video%2Fmp4&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ozone%2F20220816%2Fdummyregion%2Fs3%2Faws4_request&X-Amz-Date=20220816T164919Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=xxxxxxxxxxxxxxxxxxxxxxxxx

For more details on how to use it, please refer to:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

[adoroszlai]

There are also quite a few checkstyle violations:

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
 369: Line is longer than 80 characters (found 81).
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java
 55: Name 'ContentType' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 56: Name 'ContentLanguage' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 57: Name 'Expires' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 58: Name 'CacheControl' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 59: Name 'ContentDisposition' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 60: Name 'ContentEncoding' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 62: Variable 'headers' must be private and have accessor methods.
 63: Variable 'rest' must be private and have accessor methods.
 64: Variable 'client' must be private and have accessor methods.
 65: Variable 'body' must be private and have accessor methods.
 130: 'body' hides a field.
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestOverrideResponseHeader.java
 45: Missing a Javadoc comment.
 48: Name 'ContentType1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 49: Name 'ContentType2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 50: Name 'ContentLanguage1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 51: Name 'ContentLanguage2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 52: Name 'Expires1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 53: Name 'Expires2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 54: Name 'CacheControl1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 55: Name 'CacheControl2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 56: Name 'ContentDisposition1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 57: Name 'ContentDisposition2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 59: Name 'ContentEncoding1' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 60: Name 'ContentEncoding2' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
 62: Variable 'headers' must be private and have accessor methods.
 63: Variable 'context' must be private and have accessor methods.
 64: Variable 'rest' must be private and have accessor methods.
 85: Line is longer than 80 characters (found 83).
 107: Line is longer than 80 characters (found 83).
 114: Line is longer than 80 characters (found 82).

Can you please enable the "build-branch" workflow in your fork?

[adoroszlai]

@xichen01 Can you please check unit test failure? I think it needs mock for the new context variable.

Error:  Tests run: 31, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.044 s <<< FAILURE! - in org.apache.hadoop.ozone.s3.metrics.TestS3GatewayMetrics
Error:  org.apache.hadoop.ozone.s3.metrics.TestS3GatewayMetrics.testGetKeySuccess  Time elapsed: 0.022 s  <<< ERROR!
java.lang.NullPointerException
	at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.get(ObjectEndpoint.java:364)
	at org.apache.hadoop.ozone.s3.metrics.TestS3GatewayMetrics.testGetKeySuccess(TestS3GatewayMetrics.java:366)

[adoroszlai]

@GeorgeJahad @kerneltime @kuenishi would you like to take a look?

[GeorgeJahad]

Thanks for asking, @adoroszlai , but I'm too busy to look at this right now.