[Question]: An error occurs when Redis is deployed locally

[1364909601]

Self Checks

• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (Language Policy).
• [x] Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• [x] Please do not modify this template :) and fill in all the required fields.

Describe your problem

I also encountered this problem when deploying locally. The version of RAGFlow is v0.17.2-slim.

1. Use docker compose -f docker-compose-base.yml up -d to start all components except ragflow. After task_executor.py is started locally, the following error message is displayed: RedisDB.queue_info rag_flow_svr_queue got exception: no such key

2. Run docker compose -f docker-compose.yml up -d to start ragflow normally

To solve the problem of RedisDB.queue_info rag_flow_svr_queue got exception: no such key, I searched on the network and found the following solutions:

1. Upgrade Mysql or Ragflow
2. Restart es and redis docker images constantly
3. Access the redis docker image and manually add the RedisDB.queue value

However, I carried out all the above schemes, but did not solve the problem.

The same problem has been raised, but not solved, in ragflow issue: https://github.com/infiniflow/ragflow/issues/4257 https://github.com/infiniflow/ragflow/issues/4197 https://github.com/infiniflow/ragflow/issues/4996 https://github.com/infiniflow/ragflow/issues/5365 https://github.com/infiniflow/ragflow/issues/6054 https://github.com/infiniflow/ragflow/issues/6454 https://github.com/infiniflow/ragflow/issues/6618

[dosubot[bot]]

Hello, @1364909601! I'm here to help you with bug fixes, answer your questions, and guide you on becoming a contributor. Let's tackle this issue together!

To resolve the "RedisDB.queue_info rag_flow_svr_queue got exception: no such key" error when deploying RAGFlow version v0.17.2-slim locally using Docker Compose, you can try the following steps:

1. Switch to the Nightly Image: Use the nightly image version of RAGFlow instead of version 0.17.2. This issue is related to recent changes in the GraphRAG codebase, and switching to the nightly image version should address the problem.

2. Cancel and Restart: If the system is stuck, cancel the current parsing jobs and restart the entity process. This can help clear any jammed processes.

3. Optimize File Processing: To prevent the issue from occurring, keep parsing files smaller than 10MB and process files in smaller batches instead of large volumes (like multiple books) at once. This can help manage the load and prevent the system from getting stuck.

Additionally, according to the latest information, the Redis connection warnings, including the "no such key" warning, can be safely ignored as they do not affect the core functionality of RAGFlow. Therefore, no further action is required to resolve these warnings ^[1][2][3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[1364909601]

Hello, @1364909601! I'm here to help you with bug fixes, answer your questions, and guide you on becoming a contributor. Let's tackle this issue together!

To resolve the "RedisDB.queue_info rag_flow_svr_queue got exception: no such key" error when deploying RAGFlow version v0.17.2-slim locally using Docker Compose, you can try the following steps:

1. Switch to the Nightly Image: Use the nightly image version of RAGFlow instead of version 0.17.2. This issue is related to recent changes in the GraphRAG codebase, and switching to the nightly image version should address the problem.
2. Cancel and Restart: If the system is stuck, cancel the current parsing jobs and restart the entity process. This can help clear any jammed processes.
3. Optimize File Processing: To prevent the issue from occurring, keep parsing files smaller than 10MB and process files in smaller batches instead of large volumes (like multiple books) at once. This can help manage the load and prevent the system from getting stuck.

Additionally, according to the latest information, the Redis connection warnings, including the "no such key" warning, can be safely ignored as they do not affect the core functionality of RAGFlow. Therefore, no further action is required to resolve these warnings [1][2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

I am not currently parsing the file, I am running into a RedisDB error when I start task_executor.py on the local deployment.

[KevinHuSh]

exception: no such key are caused by dity data which is generated by running task_executor.py up and down. You could ignore that message.

[juancaoviedo]

There are multiple issues rising the problem of the term.freq file. All answers are the same: "ignore warnings".

However, there are multiple problems related to Redis. Likely people is linking the term.freq file to the other Redis problems. So, I tried to tackle every problem on isolated way.

WARNING 42117 Load term.freq FAIL!

This warning is triggered on the rag/nlp/term_weight.py file on the __init__ of the class Dealer.

try:
    self.df = load_dict(os.path.join(fnm, "term.freq"))
except Exception:
    logging.warning("Load term.freq FAIL!")

To solve this warning, I added a dummy term.freq file on the rag/res folder to avoid the warning. The file looks like this:

term1\t10
term2\t20

Suggested fix for the developer team: If the developer team do not want to provide the file because is private, maybe, providing a dummy file to avoid the warning might be good, thanks.

WARNING 42117 Realtime synonym is disabled, since no redis connection.

This warning comes from rag/nlp/synonym.py file, Dealer class. There are two more Dealer classes on rag/nlp/term_weight.py and rag/nlp/search.py. The Dealer class on rag/nlp/synonym.py that creates the warning receives as input redis = None. After, in line 39 says:

if not redis:
    logging.warning(
        "Realtime synonym is disabled, since no redis connection.")

So, if the class is instantiated without the redis argument, that warning will be triggered always. This class is instantiated on the rag/nlp/query.py file on the class FulltextQueryer as: self.syn = synonym.Dealer(), without redis, so, the warning will always be triggered.

Suggested fix for the developer team: Maybe avoiding the warning to always be triggered might be good.

RedisDB.queue_info rag_flow_svr_queue got exception: no such key

This warning is triggered on rag/utils/redis_conn.py line 315 on method queue_info.

def queue_info(self, queue, group_name) -> dict | None:
    try:
        groups = self.REDIS.xinfo_groups(queue)
        for group in groups:
            if group["name"] == group_name:
                return group
    except Exception as e:
        logging.warning(
            "RedisDB.queue_info " + str(queue) + " got exception: " + str(e)
        )
    return None

The method queue_info is called on the rag/svr/task_executor.py file on the method report_status. My guess is that the report_status is simply reporting the status and after initialization there are no queues create yet. Likely there is another process that creates those queues after.

Suggested fix for the developer team: Maybe change the logger.warning for a logger.info and handle the warning with an error in the case that the queue is required and it does not exist.

I do not understand the system good enough yet, so, my suggestions lack a lot of context. Take them with precaution. And, in the end, it seems you can just ignore all those warnings, as the developers already said.

Kudos to the developer team. Ragflow it's an Amazing project!

[1364909601]

There are multiple issues rising the problem of the term.freq file. All answers are the same: "ignore warnings".

However, there are multiple problems related to Redis. Likely people is linking the term.freq file to the other Redis problems. So, I tried to tackle every problem on isolated way.

WARNING 42117 Load term.freq FAIL!

This warning is triggered on the rag/nlp/term_weight.py file on the init of the class "Dealer".

try: self.df = load_dict(os.path.join(fnm, "term.freq")) except Exception: logging.warning("Load term.freq FAIL!") To solve this warning, I added a dummy term.fre file on the rag/res folder to avoid the warning. The file looks like this:

term1\t10
term2\t20

Suggested fix for the developer team: If the developer team do not want to provide the file because is private, maybe, providing a dummy file to avoid the warning might be good, thanks.

WARNING 42117 Realtime synonym is disabled, since no redis connection.

This warning comes from rag/nlp/synonym.py file, "Dealer" class. There are two more "Dealer" classes on rag/nlp/term_weight.py and rag/nlp/search.py. The Dealer class on rag/nlp/synonym.py that creates the warning receives as input "redis = None". After. in line 39 says:

if not redis: logging.warning( "Realtime synonym is disabled, since no redis connection.") So, if the class is instantiated without the redis argument, that warning will be triggered always. This class is instantiated on the rag/nlp/query.py file on the class "FulltextQueryer" as: self.syn = synonym.Dealer(), without redis, so, the warning will always be triggered.

Suggested fix for the developer team: Maybe avoiding the warning to always be triggered might be good.

RedisDB.queue_info rag_flow_svr_queue got exception: no such key

This warning is triggered on rag/utils/redis_conn.py line 315 on method "queue_info".

def queue_info(self, queue, group_name) -> dict | None: try: groups = self.REDIS.xinfo_groups(queue) for group in groups: if group["name"] == group_name: return group except Exception as e: logging.warning( "RedisDB.queue_info " + str(queue) + " got exception: " + str(e) ) return None The method queue_info is called on the rag/svr/task_executor.py file on the method "report_status". My guess is that the "report_status" is simply reporting the status and after initialization there are no queues create yet. Likely there is another process that creates those queues after. Suggested fix for the developer team: Maybe change the logger.warning for a logger.info and handle the warning with an error in the case that the queue is required and it is not existent.

I do not understand the system good enough yet, so, my suggestions lack a lot of context. Take them with precaution. And, in the end, it seems you can just ignore all those warnings, as the developers already said.

Kudos to the developer team. Ragflow it's an Amazing project!

Yes, I ignore this information and continue to execute task_executor.py and ragflow_server.py. Then, by switching to start the front end under the web, Ragflow can be deployed locally. However, the more significant problem encountered at present is that some third-party Python libraries cannot be compiled when migrated to other platforms, resulting in the inability to start locally.

[newforesee]

There are multiple issues rising the problem of the term.freq file. All answers are the same: "ignore warnings".

However, there are multiple problems related to Redis. Likely people is linking the term.freq file to the other Redis problems. So, I tried to tackle every problem on isolated way.

WARNING 42117 Load term.freq FAIL!

This warning is triggered on the rag/nlp/term_weight.py file on the __init__ of the class Dealer.

try: self.df = load_dict(os.path.join(fnm, "term.freq")) except Exception: logging.warning("Load term.freq FAIL!") To solve this warning, I added a dummy term.freq file on the rag/res folder to avoid the warning. The file looks like this:

term1\t10
term2\t20

Suggested fix for the developer team: If the developer team do not want to provide the file because is private, maybe, providing a dummy file to avoid the warning might be good, thanks.

WARNING 42117 Realtime synonym is disabled, since no redis connection.

This warning comes from rag/nlp/synonym.py file, Dealer class. There are two more Dealer classes on rag/nlp/term_weight.py and rag/nlp/search.py. The Dealer class on rag/nlp/synonym.py that creates the warning receives as input redis = None. After, in line 39 says:

if not redis: logging.warning( "Realtime synonym is disabled, since no redis connection.") So, if the class is instantiated without the redis argument, that warning will be triggered always. This class is instantiated on the rag/nlp/query.py file on the class FulltextQueryer as: self.syn = synonym.Dealer(), without redis, so, the warning will always be triggered.

Suggested fix for the developer team: Maybe avoiding the warning to always be triggered might be good.

RedisDB.queue_info rag_flow_svr_queue got exception: no such key

This warning is triggered on rag/utils/redis_conn.py line 315 on method queue_info.

def queue_info(self, queue, group_name) -> dict | None: try: groups = self.REDIS.xinfo_groups(queue) for group in groups: if group["name"] == group_name: return group except Exception as e: logging.warning( "RedisDB.queue_info " + str(queue) + " got exception: " + str(e) ) return None The method queue_info is called on the rag/svr/task_executor.py file on the method report_status. My guess is that the report_status is simply reporting the status and after initialization there are no queues create yet. Likely there is another process that creates those queues after.

Suggested fix for the developer team: Maybe change the logger.warning for a logger.info and handle the warning with an error in the case that the queue is required and it does not exist.

I do not understand the system good enough yet, so, my suggestions lack a lot of context. Take them with precaution. And, in the end, it seems you can just ignore all those warnings, as the developers already said.

Kudos to the developer team. Ragflow it's an Amazing project!

@juancaoviedo Thanks for your sharing. Very thorough analysis, I'm also looking into why I clearly configured Redis but prompted with Realtime synonym is disabled, since no redis connection. when I see None passed in for instantiation I do have a lot of confusion, it seems like some of the functionality is outside of the open source branch? Or maybe some features are private and not open to the open source version.

[juancaoviedo]

@newforesee yeah, it's confusing and I do not have the answer. However, as the developers already said, you can ommit the warning and the platform will work fine. I know it's not nice to see the warning as soon as you are starting with the development and you might think that is a bug, but no, it is a warning and the product can work with that warning perfectly fine. Moreover, as you advance you will see that the product is very good. So, you can just ignore the warning. However, you need to be sure that the redis connection is actually stablished. There are multiple files where the redis host, port and password are mentioned when deploying and testing locally. So, be sure that all files share same configuration for host, port, and password. As long as the services are able to talk to each other, the system will properly work.

[colin4k]

@newforesee yeah, it's confusing and I do not have the answer. However, as the developers already said, you can ommit the warning and the platform will work fine. I know it's not nice to see the warning as soon as you are starting with the development and you might think that is a bug, but no, it is a warning and the product can work with that warning perfectly fine. Moreover, as you advance you will see that the product is very good. So, you can just ignore the warning. However, you need to be sure that the redis connection is actually stablished. There are multiple files where the redis host, port and password are mentioned when deploying and testing locally. So, be sure that all files share same configuration for host, port, and password. As long as the services are able to talk to each other, the system will properly work.

Cannot ignore this warning, it will produce very huge log file unit it uses all the hardisk's space.

[Zha-Miku]

I encountered this error during file upload

[alihamza1214]

I also encountered this error and its because of "rag_flow_svr_queue" does not exist. This error occurs when i try to upload the file and start the parsing of file.

[alihamza1214]

I also encountered this error and its because of "rag_flow_svr_queue" does not exist. This error occurs when i try to upload the file and start the parsing of file.

Fixed this, actually entry_point.sh is not executing the task_executor.py so you either have to run it manually:

JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so;
LD_PRELOAD=$JEMALLOC_PATH python rag/svr/task_executor.py 1;

if you get "api" module not found then fix the path:

export PYTHONPATH=<project-absolute-path-here>:$PYTHONPATH

e.g export PYTHONPATH=/Users/abc/PycharmProjects/ragflow:$PYTHONPATH

or fix the entry_point.sh by replacing the original task_exec function with this

`function task_exe() { local consumer_id="$1" local host_id="$2" local task_id="${host_id}_${consumer_id}"

echo "[$(date)] task_exe: Starting task executor with consumer_id=$consumer_id, host_id=$host_id"
echo "[$(date)] task_exe: Full task ID will be: $task_id"
echo "[$(date)] task_exe: Python executable: $PY"
echo "[$(date)] task_exe: Current directory: $(pwd)"
echo "[$(date)] task_exe: PYTHONPATH: $PYTHONPATH"

# Check if jemalloc exists
if command -v pkg-config >/dev/null 2>&1; then
    echo "[$(date)] task_exe: pkg-config found"
    if pkg-config --exists jemalloc 2>/dev/null; then
        JEMALLOC_PATH="$(pkg-config --variable=libdir jemalloc)/libjemalloc.so"
        echo "[$(date)] task_exe: jemalloc path: $JEMALLOC_PATH"
        if [ -f "$JEMALLOC_PATH" ]; then
            echo "[$(date)] task_exe: jemalloc library exists"
        else
            echo "[$(date)] task_exe: WARNING - jemalloc library not found at $JEMALLOC_PATH"
            JEMALLOC_PATH=""
        fi
    else
        echo "[$(date)] task_exe: WARNING - jemalloc not found via pkg-config"
        JEMALLOC_PATH=""
    fi
else
    echo "[$(date)] task_exe: WARNING - pkg-config not found, skipping jemalloc"
    JEMALLOC_PATH=""
fi

# Check if the script exists
if [ -f "rag/svr/task_executor.py" ]; then
    echo "[$(date)] task_exe: task_executor.py found"
else
    echo "[$(date)] task_exe: ERROR - task_executor.py not found at rag/svr/task_executor.py"
    echo "[$(date)] task_exe: Current directory contents:"
    ls -la
    return 1
fi

local attempt=0
while true; do
    attempt=$((attempt + 1))
    echo "[$(date)] task_exe: Starting attempt #$attempt for task executor $task_id"

    # Log the exact command being run
    if [ -n "$JEMALLOC_PATH" ]; then
        echo "[$(date)] task_exe: Running: LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $task_id"
        LD_PRELOAD="$JEMALLOC_PATH" "$PY" rag/svr/task_executor.py "$task_id" 2>&1 | while IFS= read -r line; do
            echo "[$(date)] task_exe[$task_id]: $line"
        done
        exit_code=${PIPESTATUS[0]}
    else
        echo "[$(date)] task_exe: Running: $PY rag/svr/task_executor.py $task_id"
        "$PY" rag/svr/task_executor.py "$task_id" 2>&1 | while IFS= read -r line; do
            echo "[$(date)] task_exe[$task_id]: $line"
        done
        exit_code=${PIPESTATUS[0]}
    fi

    echo "[$(date)] task_exe: Task executor $task_id exited with code $exit_code"
    echo "[$(date)] task_exe: Waiting 5 seconds before restart..."
    sleep 5
done

}`

@Zha-Miku