HDDS-10634. Recon - listKeys API for listing keys with optional filters

[devmadhuu]

This PR adds a new API in Recon for listing keys for OBS buckets, Legacy buckets and FSO buckets with filters and recursively in a flat structure for OBS, LEGACY and FSO buckets.

New API:

api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket/&limit=105

Default values of API parameters if not provided:

1. replicationType - empty string and filter will not be applied, so list out all keys irrespective of replication type.
2. creationTime - empty string and filter will not be applied, so list out keys irrespective of age, else list out keys which got created on or after provided creationTime
3. keySize - 0 bytes, which means all keys greater than zero bytes will be listed, effectively all.
4. startPrefix - /
5. prevKey - ""
6. limit - 1000 Behavior of API: For OBS bucket - list out limit number of keys on the provided path. This API will implement pagination support using prevKey and limit params.

Get List of All Keys: GET /api/v1/keys/listKeys

API params:

1. replicationType - Filter for RATIS or EC replication keys
2. creationDate in "MM-dd-yyyy HH:mm:ss" string format.
3. startPrefix
4. prevKey
5. limit
6. keySize

Now lets consider, we have following OBS, LEGACY and FSO bucket key/files namespace tree structure

For OBS Bucket

• /volume1/obs-bucket/key1
• /volume1/obs-bucket/key1/key2
• /volume1/obs-bucket/key1/key2/key3
• /volume1/obs-bucket/key4
• /volume1/obs-bucket/key5
• /volume1/obs-bucket/key6

For LEGACY Bucket

• /volume1/legacy-bucket/key
• /volume1/legacy-bucket/key1/key2
• /volume1/legacy-bucket/key1/key2/key3
• /volume1/legacy-bucket/key4
• /volume1/legacy-bucket/key5
• /volume1/legacy-bucket/key6

For FSO Bucket

• /volume1/fso-bucket/dir1/dir2/dir3
• /volume1/fso-bucket/dir1/testfile
• /volume1/fso-bucket/dir1/file1
• /volume1/fso-bucket/dir1/dir2/testfile
• /volume1/fso-bucket/dir1/dir2/file1
• /volume1/fso-bucket/dir1/dir2/dir3/testfile
• /volume1/fso-bucket/dir1/dir2/dir3/file1

Input Request for OBS bucket:

   `api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`

Output Response:

{
   "status": "OK",
   "path": "/volume1/obs-bucket",
   "replicatedDataSize": 20971520,
   "unReplicatedDataSize": 20971520,
   "keyCount": 2,
   "lastKey": "/volume1/obs-bucket/key1/key2",
   "keys": [
       {
           "key": "/volume1/obs-bucket/key1",
           "path": "key1",
           "inStateSince": 1715174266126,
           "size": 10485760,
           "replicatedSize": 10485760,
           "replicationInfo": {
               "replicationFactor": "ONE",
               "requiredNodes": 1,
               "replicationType": "RATIS"
           },
           "creationTime": 1715174266126,
           "modificationTime": 1715174267480,
           "isKey": true
       },
       {
           "key": "/volume1/obs-bucket/key1/key2",
           "path": "key1/key2",
           "inStateSince": 1715174269510,
           "size": 10485760,
           "replicatedSize": 10485760,
           "replicationInfo": {
               "replicationFactor": "ONE",
               "requiredNodes": 1,
               "replicationType": "RATIS"
           },
           "creationTime": 1715174269510,
           "modificationTime": 1715174270410,
           "isKey": true
       }
   ]
}

Input Request for FSO bucket:

       `api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`

Output Response:

{
    "status": "OK",
    "path": "/volume1/fso-bucket",
    "replicatedDataSize": 62914560,
    "unReplicatedDataSize": 20971520,
    "keyCount": 2,
    "lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
    "keys": [
        {
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
            "path": "file1",
            "inStateSince": 1715174237440,
            "size": 10485760,
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "replicationType": "RATIS"
            },
            "creationTime": 1715174237440,
            "modificationTime": 1715174238161,
            "isKey": true
        },
        {
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
            "path": "testfile",
            "inStateSince": 1715174234840,
            "size": 10485760,
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "replicationType": "RATIS"
            },
            "creationTime": 1715174234840,
            "modificationTime": 1715174235562,
            "isKey": true
        }
    ]
}

Input Request for Legacy bucket:

       `api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`

Output Response:

{
    "status": "OK",
    "path": "/volume1/legacy-bucket",
    "replicatedDataSize": 52428800,
    "unReplicatedDataSize": 52428800,
    "keyCount": 2,
    "lastKey": "/volume1/legacy-bucket/key1/key2",
    "keys": [
        {
            "key": "/volume1/legacy-bucket/key1",
            "path": "key1",
            "inStateSince": 1715174303702,
            "size": 10485760,
            "replicatedSize": 10485760,
            "replicationInfo": {
                "replicationFactor": "ONE",
                "requiredNodes": 1,
                "replicationType": "RATIS"
            },
            "creationTime": 1715174303702,
            "modificationTime": 1715174304619,
            "isKey": true
        },
        {
            "key": "/volume1/legacy-bucket/key1/key2",
            "path": "key1/key2",
            "inStateSince": 1715174306641,
            "size": 41943040,
            "replicatedSize": 41943040,
            "replicationInfo": {
                "replicationFactor": "ONE",
                "requiredNodes": 1,
                "replicationType": "RATIS"
            },
            "creationTime": 1715174306641,
            "modificationTime": 1715174307994,
            "isKey": true
        }
    ]
}

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10634

How was this patch tested?

Added Junit test cases and tested various assertions.

[devmadhuu]

@sodonnel @sumitagrawl @ArafatKhan2198 @dombizita kindly review.