dify icon indicating copy to clipboard operation
dify copied to clipboard

feat: When using redis cluster for caching, add global_prefix to all keys cached by redis

Open jiandanfeng opened this issue 10 months ago • 2 comments

Summary

Fixes #13604 Add REDIS_KEY_PREFIX configuration, default is empty

[!Tip] Close issue syntax: Fixes #<issue number> or Resolves #<issue number>, see documentation for more details.

Screenshots

Before After
... image

Checklist

[!IMPORTANT]
Please review the checklist below before submitting your pull request.

  • [ ] This change requires a documentation update, included: Dify Document
  • [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • [x] I've updated the documentation accordingly.
  • [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

jiandanfeng avatar Feb 13 '25 05:02 jiandanfeng

After using these changes, when my Redis cluster is version 6.0 and ACL user management is enabled, I encounter some issues. For example, the Redis cluster must enter the account and password at the same time to log in. At the same time, an error occurs when executing the redis_client.lock code, indicating that the current user has no permission to execute the command. I have solved the relevant issues. How can I submit the solution?

berg-wang avatar Mar 26 '25 03:03 berg-wang

my solution is : 1、/api/extensions/ext_redis.py

 elif dify_config.REDIS_USE_CLUSTERS:
        assert dify_config.REDIS_CLUSTERS is not None, "REDIS_CLUSTERS must be set when REDIS_USE_CLUSTERS is True"
        nodes = [
            ClusterNode(host=node.split(":")[0], port=int(node.split(":")[1]))
            for node in dify_config.REDIS_CLUSTERS.split(",")
        ]
        # FIXME: mypy error here, try to figure out how to fix it
        username=dify_config.REDIS_CLUSTERS_USERNAME
        if username:
            redis_client.initialize(
                RedisCluster(startup_nodes=nodes, password=dify_config.REDIS_CLUSTERS_PASSWORD, username=dify_config.REDIS_CLUSTERS_USERNAME),
                prefix=dify_config.REDIS_KEY_PREFIX,
            )  # type: ignore
        else:
            redis_client.initialize(
                RedisCluster(startup_nodes=nodes, password=dify_config.REDIS_CLUSTERS_PASSWORD),
                prefix=dify_config.REDIS_KEY_PREFIX,
            )  # type: ignore

2、/api/commands.py

@click.command("upgrade-db", help="Upgrade the database")
def upgrade_db():
    click.echo("Preparing database migration...")
    lock = redis_client.lock(name=f"{dify_config.REDIS_KEY_PREFIX}:db_upgrade_lock", timeout=60)
    if lock.acquire(blocking=False):

3、/api/services/dataset_service.py there are four positions that need to change in this py-file

 lock_name = "{}:add_document_lock_dataset_id_{}".format(dify_config.REDIS_KEY_PREFIX, dataset.id)
            with redis_client.lock(lock_name, timeout=600):

berg-wang avatar Mar 26 '25 03:03 berg-wang

Hello sorry for the late response, could you resolve the conflicts and trigger it again?

crazywoola avatar Jul 15 '25 06:07 crazywoola

@jiandanfeng Hello could you resolve the conflict again?

crazywoola avatar Aug 29 '25 09:08 crazywoola