[Feature] Add Redis Catalog

[zhangbutao]

Feature request

Is your feature request related to a problem? Please describe.

This Redis catalog feature is inspired by Trino's Redis catalog. Users of the Trino Redis catalog can easily migrate to the SR Redis catalog. Users can conduct joint queries across multiple data sources based on multi-catalog, such as hive/iceberg/Redis catalog.

Prototype Design

When initially designing, the Java client of Redis was considered to be more mature and stable. Therefore, it was decided to use the JNI method at the BE layer to read Redis data, while the FE layer was responsible for managing Redis metadata, generating execution plans, etc. This solution is similar to the current JDBC catalog of SR. Especially for the JNI reading process on the BE side, the CPP JNI code part of the Redis catalog referred to the CPP JNI implementation of JDBC; however, Redis does not have the concept of structured metadata like JDBC, so the metadata of Redis can only be customized by users in the form of a JSON file, and then the FE reads the JSON-defined Redis metadata information. The JSON definition process of the metadata of the Redis catalog referred to the implementation of Trino Redis catalog.

NOTE: The key definition in Redis must follow the format of dbName:tblName:xx. StarRocks will retrieve all keys and their values that match dbName:tblName:*, and then return these keys/values in the form of StarRocks table data. For example, if a user sets Redis keys as testdb:test:aa and testdb:test:bb, these two key/value pairs belong to the same database testdb and the test table. Users can query the Redis data by executing select * from testdb.test.

FE side Redis catalog design

The implementation of FE mainly focuses on how to parse the user-defined JSON files to construct a structured Redis metadata structure, and send the constructed table information (database table name, column name) to BE. Other parts of the code, such as PhysicalRedisScanOperator.java, are basically consistent with the existing other catalog code implementations.

You can use the following command to create the Redis catalog:

CREATE EXTERNAL CATALOG redis_catalog
PROPERTIES
(
    "type"="redis",
    "password"="pawssd",
    "redis_uri"="127.0.0.1:6379",
    "redis.table-description-dir"="/path/redis"
);

Detailed usage examples can be found in the Doc section of the PR Add Redis Catalog.

BE side Redis catalog design

On the BE side, Redis data is read using the JNI method. The entire process of JNI reading is based on the implementation of the jdbc catalog. The following is an illustration of the code flow for reading Redis data from the Redis catalog on the BE side:

Figure 2. The Redis-related content in the query_rpc thread mainly involves initializing RedisDataSourceProvider, which contains the scan node information of Redis and is used for subsequent initialization of RedisDataSource. RedisDataSource is the main class for performing Redis read operations.

Figure 3. The pip_exec_com thread mainly accomplishes the construction of the RedisDataSource object, as well as the subsequent construction of RedisScanner for the RedisDataSource user.

Figure 4.The pip_con_scan_co thread mainly initializes the RedisScanner using the RedisDataSource object. Within the RedisScanner, it uses JNI to interact with the Redis service, scans the Redis data, and encapsulates it into StarRocks data blocks before returning them to the client. The execution process of JNI is generally similar to that of the jdbc catalog.

[alvin-celerdata]

@zhangbutao I think it will be better to have a google doc to review and comment, could you create one?

[zhangbutao]

@zhangbutao I think it will be better to have a google doc to review and comment, could you create one?

Hi @alvin-celerdata. I have created a google doc about StarRocks Redis catalog design, please check.

Please let me know if you need any more information. Thank you.

[alvin-celerdata]

@zhangbutao, I left some comments in the documents. Could you take a look?

[zhangbutao]

@zhangbutao, I left some comments in the documents. Could you take a look?

@alvin-celerdata Sorry for the late reply. I replied to your question on the doc. If you have any other questions, please feel free to ask.

Thank you very much for your time!

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!