[Feature] Add Redis Catalog

[zhangbutao]

Why I'm doing:

We want to query redis datasource through SR, and do some federated query with other datasource, such as iceberg&hive&jdbc table. Some other community users also want to this feature https://github.com/StarRocks/starrocks/issues/56403

This PR added redis catalog, and the implementation principle is based on that of Trion redis catalog. Therefore, users can use the SR query to access Redis data just as they do with Trino.

Fixes https://github.com/StarRocks/starrocks/issues/59842

What I'm doing:

This PR used JNI to fetch redis data, just like jdbc catalog. Perhaps later I will attempt to use the Redis client for C++ to optimize the code. But we have been using the JNI method for some time now and it has worked well. Therefore, I plan to continue using the JNI approach at this stage.

There are still some aspects that need to be further improved in the subsequent PR: 1: Added more redis data type 2: Added internal column like trino redis catalog 3: Added redis acl auth

A simple test demo:

• 1: create redis key/value through redis client:

set testdb:test:bb "{\"rowNumber\":2,\"customerKey\":2,\"name\":\"Customer#000000002\"}"
set testdb:test:aa "{\"rowNumber\":1,\"customerKey\":1,\"name\":\"Customer#000000001\"}"

• 2: Create sr redis json file like redis_schema.json & you can create multiple json file based on your redis data structure. And then put this file in FE local directory like /opt/redis. The following json file is same as Trino's redis json file.

{
   "tableName": "test",
   "schemaName": "testdb",
   "key": {
       "dataFormat": "raw",
       "fields": [
            {
                "name":"redis_key",
                "type":"VARCHAR"
           }
       ]
   },
   "value": {
       "dataFormat": "json",
       "fields": [
           {
                "name":"row_number",
                "type":"BIGINT"
           },
           {
                "name":"customer_key",
                "type":"BIGINT"
           },
           {
                "name":"name",
                "type":"VARCHAR"
           }
      ]
    }
}

• 3: Create redis catalog through SR

CREATE EXTERNAL CATALOG redis_catalog
PROPERTIES
(
    "type"="redis",
    "password"="passwd",
    "redis_uri"="redis_ip:port",
    "redis.table-description-dir"="/opt/redis"
);

• 4: Query redis data through SR

mysql> select * from redis_catalog.testdb.test;
+----------------+------------+--------------+--------------------+
| redis_key      | row_number | customer_key | name               |
+----------------+------------+--------------+--------------------+
| testdb:test:bb | 2          |            2 | Customer#000000002 |
| testdb:test:aa | 1          |            1 | Customer#000000001 |
+----------------+------------+--------------+--------------------+
2 rows in set (1.44 sec)

Fixes #issue

What type of PR is this:

• [ ] BugFix
• [x] Feature
• [ ] Enhancement
• [ ] Refactor
• [ ] UT
• [ ] Doc
• [ ] Tool

Does this PR entail a change in behavior?

• [ ] Yes, this PR will result in a change in behavior.
• [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
• [ ] Parameter changes: default values, similar parameters but with different default values
• [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
• [ ] Feature removed
• [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• [ ] I have added test cases for my bug fix or my new feature
• [ ] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function
• [ ] This is a backport pr

Bugfix cherry-pick branch check:

• [ ] I have checked the version labels which the pr will be auto-backported to the target branch
  • [ ] 3.5
  • [ ] 3.4
  • [ ] 3.3
  • [ ] 3.2
  • [ ] 3.1

[zhangbutao]

@zhangbutao Thank you for the contribution. I saw the demo description of this feature, that is good. However, I think it is better to explain the design of technical of this feature in the GitHub ticket. Because this will help others understand the logic and details of your implementing from a higher scope. This change has about 2000 lines of code, it is not easy for others to review it without understanding the design details.

The design doc should include

1. Overall architecture of this feature
2. The key process of this feature, such as how the data flow looks like
3. The key classes and the consideration about them
4. How this feature will influence other parts
5. handling of unexpected situations
6. other considerations
7. Necessary English documentary for this new feature, otherwise users won't know this feature and how to use it.

@alvin-celerdata Thanks for you advice! Will try to give more context about this PR in the next few days. :)

[zhangbutao]

Hi @alvin-celerdata , I added docs and issues https://github.com/StarRocks/starrocks/issues/59842. Please help to review this PR. Thanks in advance!

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
4.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[alvin-celerdata]

@cursor review

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
3.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[github-actions[bot]]

[Java-Extensions Incremental Coverage Report]

:x: fail : 0 / 132 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	com/starrocks/redis/decoder/RedisDataType.java	0	2	00.00%	[20, 21]
:large_blue_circle:	com/starrocks/redis/decoder/JsonRowDataDecoder.java	0	9	00.00%	[26, 27, 31, 32, 34, 35, 36, 37, 39]
:large_blue_circle:	com/starrocks/redis/reader/RedisScanner.java	0	48	00.00%	[35, 39, 40, 43, 45, 46, 47, 50, 51, 52, 54, 55, 57, 58, 59, 60, 61, 62, 63, 65, 68, 71, 75, 80, 81, 82, 84, 85, 86, 87, 88, 89, 92, 95, 96, 97, 98, 99, 100, 101, 103, 106, 107, 108, 110, 114, 118, 119]
:large_blue_circle:	com/starrocks/redis/decoder/CsvRowDataDecoder.java	0	5	00.00%	[24, 28, 29, 31, 32]
:large_blue_circle:	com/starrocks/redis/decoder/RowDataDecoderFactory.java	0	10	00.00%	[23, 24, 27, 28, 29, 30, 33, 34, 35, 37]
:large_blue_circle:	com/starrocks/redis/decoder/RowDataDecoder.java	0	2	00.00%	[24, 28]
:large_blue_circle:	com/starrocks/redis/reader/RedisScanContext.java	0	14	00.00%	[30, 31, 35, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
:large_blue_circle:	com/starrocks/redis/decoder/RedisDataDecodeUtil.java	0	35	00.00%	[29, 36, 37, 40, 41, 43, 44, 47, 55, 56, 59, 60, 62, 63, 66, 71, 72, 74, 75, 76, 78, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 93, 97, 99, 101]
:large_blue_circle:	com/starrocks/redis/decoder/HashRowDataDecoder.java	0	7	00.00%	[23, 27, 28, 30, 31, 32, 33]

[github-actions[bot]]

[FE Incremental Coverage Report]

:x: fail : 216 / 312 (69.23%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	com/starrocks/sql/optimizer/transformer/RelationTransformer.java	0	2	00.00%	[744, 745]
:large_blue_circle:	com/starrocks/sql/optimizer/operator/OperatorVisitor.java	0	2	00.00%	[246, 365]
:large_blue_circle:	com/starrocks/sql/optimizer/statistics/StatisticsCalculator.java	0	1	00.00%	[859]
:large_blue_circle:	com/starrocks/sql/optimizer/OptExpressionVisitor.java	0	1	00.00%	[166]
:large_blue_circle:	com/starrocks/connector/redis/RedisTableFieldDescription.java	12	37	32.43%	[70, 75, 80, 85, 90, 95, 100, 101, 103, 104, 107, 108, 109, 110, 111, 112, 113, 118, 119, 120, 121, 122, 123, 124, 125]
:large_blue_circle:	com/starrocks/catalog/RedisTable.java	11	24	45.83%	[51, 52, 54, 59, 65, 66, 67, 68, 69, 71, 72, 73, 74]
:large_blue_circle:	com/starrocks/connector/redis/RedisTableFieldGroup.java	8	15	53.33%	[44, 55, 65, 66, 67, 68, 69]
:large_blue_circle:	com/starrocks/connector/redis/RedisTableDescription.java	11	17	64.71%	[68, 69, 70, 71, 72, 73]
:large_blue_circle:	com/starrocks/sql/optimizer/operator/physical/PhysicalRedisScanOperator.java	11	16	68.75%	[30, 31, 45, 46, 50]
:large_blue_circle:	com/starrocks/planner/RedisScanNode.java	33	48	68.75%	[53, 58, 59, 69, 76, 82, 103, 104, 105, 106, 107, 108, 109, 110, 111]
:large_blue_circle:	com/starrocks/connector/redis/RedisMetadata.java	48	61	78.69%	[66, 67, 83, 85, 86, 98, 99, 118, 119, 126, 127, 132, 143]
:large_blue_circle:	com/starrocks/connector/redis/RedisConnector.java	15	18	83.33%	[50, 51, 52]
:large_blue_circle:	com/starrocks/connector/redis/RedisUtil.java	13	15	86.67%	[24, 49]
:large_blue_circle:	com/starrocks/sql/optimizer/operator/logical/LogicalRedisScanOperator.java	9	10	90.00%	[48]
:large_blue_circle:	com/starrocks/catalog/Table.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/sql/optimizer/rule/implementation/RedisScanImplementationRule.java	6	6	100.00%	[]
:large_blue_circle:	com/starrocks/sql/optimizer/rule/transformation/MergeLimitDirectRule.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/sql/optimizer/rule/RuleType.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/connector/ConnectorType.java	2	2	100.00%	[]
:large_blue_circle:	com/starrocks/sql/optimizer/operator/pattern/MultiOpPattern.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/sql/optimizer/operator/OperatorType.java	2	2	100.00%	[]
:large_blue_circle:	com/starrocks/sql/plan/PlanFragmentBuilder.java	31	31	100.00%	[]

[github-actions[bot]]

[BE Incremental Coverage Report]

:x: fail : 3 / 298 (01.01%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	be/src/exec/redis_scanner.cpp	0	199	00.00%	[44, 45, 47, 49, 51, 53, 55, 58, 59, 60, 61, 62, 63, 65, 66, 67, 68, 69, 70, 71, 73, 74, 75, 77, 78, 81, 83, 85, 86, 87, 89, 90, 91, 92, 94, 95, 99, 100, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 117, 118, 120, 121, 122, 123, 124, 126, 127, 129, 130, 131, 135, 136, 140, 141, 145, 146, 150, 151, 152, 153, 159, 160, 161, 163, 164, 165, 166, 167, 168, 169, 170, 171, 173, 174, 175, 177, 178, 179, 181, 182, 184, 185, 186, 189, 190, 192, 193, 195, 196, 198, 199, 200, 201, 202, 203, 205, 206, 209, 210, 212, 213, 214, 216, 217, 218, 220, 221, 222, 224, 225, 226, 227, 228, 229, 230, 232, 233, 234, 235, 236, 238, 239, 240, 241, 243, 244, 246, 247, 248, 249, 250, 251, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 265, 266, 267, 268, 270, 271, 273, 274, 275, 276, 279, 280, 282, 283, 284, 285, 287, 288, 290, 291, 292, 293, 294, 295, 296, 297, 298, 301, 302, 304, 305, 308, 309, 310, 311, 312, 313, 314, 315, 316, 318, 320, 321, 322]
:large_blue_circle:	be/src/connector/redis_connector.cpp	0	58	00.00%	[24, 26, 31, 32, 34, 35, 38, 39, 44, 45, 47, 48, 51, 52, 53, 54, 55, 56, 59, 60, 61, 63, 65, 66, 67, 68, 69, 70, 71, 73, 74, 75, 77, 78, 79, 82, 83, 85, 86, 88, 89, 91, 93, 96, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 108, 110, 111, 112]
:large_blue_circle:	be/src/runtime/descriptors.cpp	0	15	00.00%	[538, 540, 541, 542, 543, 544, 545, 547, 548, 549, 550, 551, 552, 782, 783]
:large_blue_circle:	be/src/exec/exec_node.cpp	0	9	00.00%	[575, 576, 577, 578, 579, 580, 581, 582, 858]
:large_blue_circle:	be/src/exec/redis_scanner.h	0	5	00.00%	[35, 47, 56, 57, 59]
:large_blue_circle:	be/src/runtime/descriptors.h	0	4	00.00%	[452, 453, 454, 455]
:large_blue_circle:	be/src/connector/redis_connector.h	2	7	28.57%	[33, 41, 46, 47, 57]
:large_blue_circle:	be/src/connector/connector.cpp	1	1	100.00%	[]

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
3.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[alvin-celerdata]

@cursor review