HDDS-7791. Support key ownership

[xichen01]

What changes were proposed in this pull request?

current support persists the owner info to DB and reads the owner info from DB,

the owner will be the user that creates the Object, for ozone fs and ozone sh key, it will be the system user that executes the command, for aws s3 it will be AWS Access Key ID

supported command:

write:

support ozone fs and ozone sh key and aws s3/s3api related Object creation commands. will present owner info to DB when creating an object.

read (support display real owner info):

if the owner field is null will fall back to the old logic, use the OS login user to fill the owner field. This is possible during an upgrade from the previous version

ozone fs -ls 
ozone fs stat
ozone sh key ls
ozone sh key info
aws s3api list-objects

aws s3api get-object-acl not support current, will be support next

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7791

How was this patch tested?

1. create some key/file/dir through ozone fs, ozone sh key, aws s3 environment

[pony.chen@linux /root/ozone]% aws configure
AWS Access Key ID [****************chen]:
// xxxxxx
[pony.chen@linux /root/ozone]% whoami
pony.chen

[pony.chen@linux /root/ozone]% ozone sh key put /s3v/bucket1/putfile ~/testfile.img
[pony.chen@linux /root/ozone]% ozone fs -mkdir ofs://localhost/s3v/bucket1/dir1
[pony.chen@linux /root/ozone]% ozone fs -touch ofs://localhost/s3v/bucket1/file1
[pony.chen@linux /root/ozone]% aws s3 --endpoint http://localhost:9878 cp ~/testfile s3://bucket1/s3file
upload: ../../testfile to s3://bucket1/s3file
[pony.chen@linux /root/ozone]%

2. switch to another user and display key/file info about owner, the owner info will be real info created above environment. You can see that the owner is pony.chen instead of root (before this PR it will be root)

linux:~ root# whoami
root

linux:~ root# ozone sh key ls s3v/bucket1 | grep "name\|owner"
  "name" : "dir1/",
  "ownerName" : "pony.chen",
  "name" : "file1",
  "ownerName" : "pony.chen",
  "name" : "putfile",
  "ownerName" : "pony.chen",
  "name" : "s3file",
  "ownerName" : "pony.chen",
linux:~ root#

linux:~ root# ozone fs -ls ofs://localhost/s3v/bucket1/
Found 4 items
drwxrwxrwx   - pony.chen root          0 2022-12-06 15:06 ofs://localhost/s3v/bucket1/dir1
-rw-rw-rw-   3 pony.chen root          0 2022-12-06 15:06 ofs://localhost/s3v/bucket1/file1
-rw-rw-rw-   3 pony.chen root    1048576 2022-12-06 15:05 ofs://localhost/s3v/bucket1/putfile
-rw-rw-rw-   3 pony.chen root    1048576 2022-12-06 15:07 ofs://localhost/s3v/bucket1/s3file
linux:~ root#

linux:~ root# aws s3api --endpoint http://localhost:9878 list-objects --bucket bucket1
None
CONTENTS        2022-12-06T07:06:07.468Z        dir1/   2022-12-06T07:06:07.468000+00:00        0       STANDARD
OWNER   pony.chen       pony.chen
CONTENTS        2022-12-06T07:06:22.887Z        file1   2022-12-06T07:06:22.887000+00:00        0       STANDARD
OWNER   pony.chen       pony.chen
CONTENTS        2022-12-06T07:05:39.955Z        putfile 2022-12-06T07:05:39.955000+00:00        1048576 STANDARD
OWNER   pony.chen       pony.chen
CONTENTS        2022-12-06T07:07:00.776Z        s3file  2022-12-06T07:07:00.776000+00:00        1048576 STANDARD
OWNER   pony.chen       pony.chen

[hemantk-12]

@errose28 can you please review the latest pull?

@xichen01 Can you please look into the workflow failures?

[errose28]

I just had some minor comments and questions posted here and on HDDS-7577. I probably won't have time to review this PR, at least not any time soon.

[adoroszlai]

@xichen01 Do you plan to update this PR? There are lots of conflicts, unfortunately.

[xichen01]

@adoroszlai @smengcl @errose28 PTAL.

[xichen01]

@smengcl @errose28 Is it sufficient for S3 to use the AWS Access ID directly as the Owner of the Object (one-to-one Access ID and Owner of the Object)

[adoroszlai]

@sumitagrawl please review

[smengcl]

@smengcl @errose28 Is it sufficient for S3 to use the AWS Access ID directly as the Owner of the Object (one-to-one Access ID and Owner of the Object)

@xichen01 Uh preferrably not. accessId should be mapped to a user name (similar idea to ugi.getShortUserName()) before it can be used in the owner field:

https://github.com/apache/ozone/blob/cce2f969a85323441c476aaeaf27d45b081b0c2f/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java#L417-L420

[xichen01]

@smengcl @errose28 Is it sufficient for S3 to use the AWS Access ID directly as the Owner of the Object (one-to-one Access ID and Owner of the Object)

@xichen01 Uh preferrably not. accessId should be mapped to a user name (similar idea to ugi.getShortUserName()) before it can be used in the owner field:

https://github.com/apache/ozone/blob/cce2f969a85323441c476aaeaf27d45b081b0c2f/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java#L417-L420

@smengcl. Thank for your suggestion. How to map the accessId to a user name. Are you saying we should use Displayname as the owner of the Ozone key? The AWS S3 Owner is the ID and Displayname, while the Ozone Owner is simply a string.

public class S3Owner {

  public static final S3Owner
      NOT_SUPPORTED_OWNER = new S3Owner("NOT-SUPPORTED", "Not Supported");

  @XmlElement(name = "DisplayName")
  private String displayName;

  @XmlElement(name = "ID")
  private String id;

}

[smengcl]

@smengcl @errose28 Is it sufficient for S3 to use the AWS Access ID directly as the Owner of the Object (one-to-one Access ID and Owner of the Object)

@xichen01 Uh preferrably not. accessId should be mapped to a user name (similar idea to ugi.getShortUserName()) before it can be used in the owner field: https://github.com/apache/ozone/blob/cce2f969a85323441c476aaeaf27d45b081b0c2f/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java#L417-L420

@smengcl. Thank for your suggestion. How to map the accessId to a user name. Are you saying we should use Displayname as the owner of the Ozone key? The AWS S3 Owner is the ID and Displayname, while the Ozone Owner is simply a string.

public class S3Owner {

  public static final S3Owner
      NOT_SUPPORTED_OWNER = new S3Owner("NOT-SUPPORTED", "Not Supported");

  @XmlElement(name = "DisplayName")
  private String displayName;

  @XmlElement(name = "ID")
  private String id;

}

@xichen01

@smengcl @errose28 Is it sufficient for S3 to use the AWS Access ID directly as the Owner of the Object (one-to-one Access ID and Owner of the Object)

@xichen01 Uh preferrably not. accessId should be mapped to a user name (similar idea to ugi.getShortUserName()) before it can be used in the owner field: https://github.com/apache/ozone/blob/cce2f969a85323441c476aaeaf27d45b081b0c2f/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java#L417-L420

@smengcl. Thank for your suggestion. How to map the accessId to a user name. Are you saying we should use Displayname as the owner of the Ozone key? The AWS S3 Owner is the ID and Displayname, while the Ozone Owner is simply a string.

public class S3Owner {

  public static final S3Owner
      NOT_SUPPORTED_OWNER = new S3Owner("NOT-SUPPORTED", "Not Supported");

  @XmlElement(name = "DisplayName")
  private String displayName;

  @XmlElement(name = "ID")
  private String id;

}

We can add a helper method to do the conversion. For accessId -> user name conversion there are two cases to be considered:

1. When accessId is generated with ozone s3 getsecret (not managed under an Ozone tenant), the accessId is the Kerberos principal, e.g. testuser/[email protected]. The conversion can be done with ugi.getShortUserName() (it automatically applies the conversion rules specified in hadoop.security.auth_to_local), where ugi can be created from String like this:

https://github.com/apache/ozone/blob/2ae531b0f6a069db5a46bd486bb50225a168485d/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpServer2.java#L1502-L1503

2. When accessId is generated with ozone tenant user assign (managed by an Ozone tenant), the accessId to username mapping is stored in tenantAccessIdTable table. So the conversion can be done by first getting the OmDBAccessIdInfo associated with the accessId, then getting its userPrincipal, which should already be the (short) user name we need for the owner field:

https://github.com/apache/ozone/blob/6d7ba130cf5a660780aceb773bb17d738df33905/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDBAccessIdInfo.java#L47-L50

And in order to know whether an accessId belongs to a tenant or not in the first place, we currently just use the trick of checking its existence in tenantAccessIdTable before s3SecretTable, like this:

https://github.com/apache/ozone/blob/9238be33a88f98a055ec61d432c3b33f972f4ac5/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/OMSetSecretRequest.java#L68-L74

[xichen01]

@smengcl PTAL.

[xichen01]

@smengcl PTAL. Thanks

[xichen01]

Thanks @xichen01 for the big patch. lgtm

Only concern left from my side is the one regarding snapshot diff: #4188 (comment)

Have modified.

[smengcl]

Thanks @xichen01 for merging master to this PR.

But there is build failure after the conflict resolution. Would you check?

[smengcl]

Thanks @xichen01 for working on this!

Also thanks @sumitagrawl @errose28 for the reviews.