infiniflow
[Question]: My knowledge database suddenly can't access the generated chunks
Describe your problem
The ragflow suddenly can't access the generated chunks when I click the name of the file after embedding chunks. The error shows"TTransportException("Could not connect to any of [('172.18.0.6', 23817)]")" and "(2008, 'Infinity is initing')". 1 min before I can access the chunks and now I cannot. What is going on?
Infinity is restoring. If infinity is ready, the log message 'Infinity is started in standalone mode.' will be printout.
I got the same problem, infinity service restarted itself again and again.
I got the same problem on k8s; infinity log as below:
01:59:20.490] [344] [critical] Txn ID: 1, Text: ForceCheckpointTask, Begin TS: 41578, Commit TS: 41579, State: Committed Checkpoint: catalog path: catalog/FULL.41577.json max commit ts: 41577 is full checkpoint: 1
ForceCheckpointTask
[01:59:20.490] [344] [critical] Txn ID: 16, Text: Full or delta checkpoint, Begin TS: 41580, Commit TS: 41581, State: Committed Checkpoint: catalog path: catalog/FULL.41579.json max commit ts: 41579 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 37, Text: Full or delta checkpoint, Begin TS: 41582, Commit TS: 41583, State: Committed Checkpoint: catalog path: catalog/FULL.41581.json max commit ts: 41581 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 55, Text: Full or delta checkpoint, Begin TS: 41584, Commit TS: 41585, State: Committed Checkpoint: catalog path: catalog/FULL.41583.json max commit ts: 41583 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 73, Text: Full or delta checkpoint, Begin TS: 41586, Commit TS: 41587, State: Committed Checkpoint: catalog path: catalog/FULL.41585.json max commit ts: 41585 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 91, Text: Full or delta checkpoint, Begin TS: 41588, Commit TS: 41589, State: Committed Checkpoint: catalog path: catalog/FULL.41587.json max commit ts: 41587 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 107, Text: Full or delta checkpoint, Begin TS: 41590, Commit TS: 41591, State: Committed Checkpoint: catalog path: catalog/FULL.41589.json max commit ts: 41589 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 127, Text: Full or delta checkpoint, Begin TS: 41592, Commit TS: 41593, State: Committed Checkpoint: catalog path: catalog/FULL.41591.json max commit ts: 41591 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 144, Text: Full or delta checkpoint, Begin TS: 41594, Commit TS: 41595, State: Committed Checkpoint: catalog path: catalog/FULL.41593.json max commit ts: 41593 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 159, Text: Full or delta checkpoint, Begin TS: 41596, Commit TS: 41597, State: Committed Checkpoint: catalog path: catalog/FULL.41595.json max commit ts: 41595 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 177, Text: Full or delta checkpoint, Begin TS: 41598, Commit TS: 41599, State: Committed Checkpoint: catalog path: catalog/FULL.41597.json max commit ts: 41597 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 195, Text: Full or delta checkpoint, Begin TS: 41600, Commit TS: 41601, State: Committed Checkpoint: catalog path: catalog/FULL.41599.json max commit ts: 41599 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 217, Text: Full or delta checkpoint, Begin TS: 41602, Commit TS: 41603, State: Committed Checkpoint: catalog path: catalog/FULL.41601.json max commit ts: 41601 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 231, Text: Full or delta checkpoint, Begin TS: 41604, Commit TS: 41605, State: Committed Checkpoint: catalog path: catalog/FULL.41603.json max commit ts: 41603 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 253, Text: Full or delta checkpoint, Begin TS: 41606, Commit TS: 41607, State: Committed Checkpoint: catalog path: catalog/FULL.41605.json max commit ts: 41605 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 271, Text: Full or delta checkpoint, Begin TS: 41608, Commit TS: 41609, State: Committed Checkpoint: catalog path: catalog/FULL.41607.json max commit ts: 41607 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 285, Text: Full or delta checkpoint, Begin TS: 41610, Commit TS: 41611, State: Committed Checkpoint: catalog path: catalog/FULL.41609.json max commit ts: 41609 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 303, Text: Full or delta checkpoint, Begin TS: 41612, Commit TS: 41613, State: Committed Checkpoint: catalog path: catalog/FULL.41611.json max commit ts: 41611 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 321, Text: Full or delta checkpoint, Begin TS: 41614, Commit TS: 41615, State: Committed Checkpoint: catalog path: catalog/FULL.41613.json max commit ts: 41613 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 339, Text: Full or delta checkpoint, Begin TS: 41616, Commit TS: 41617, State: Committed Checkpoint: catalog path: catalog/FULL.41615.json max commit ts: 41615 is full checkpoint: 1
[01:59:20.490] [344] [critical] Txn ID: 354, Text: Select Statement, Begin TS: 41618, Commit TS: 0, State: Started
[01:59:20.490] [344] [critical] Txn ID: 350, Text: Select Statement, Begin TS: 41618, Commit TS: 0, State: Started
[01:59:20.491] [344] [critical] Error: This task should not be scheduled if the source queue is empty@src/executor/operator_state.cpp:50 [01:59:20.491] [344] [critical] 0, /usr/bin/infinity(+0x4e0629) [0x55b80efe4629] [01:59:20.491] [344] [critical] 1, /usr/bin/infinity(+0x4e1474) [0x55b80efe5474] [01:59:20.491] [344] [critical] 2, /usr/bin/infinity(+0x13b3e6c) [0x55b80feb7e6c] [01:59:20.491] [344] [critical] 3, /usr/bin/infinity(+0x60ad70) [0x55b80f10ed70] [01:59:20.491] [344] [critical] 4, /usr/bin/infinity(+0x2be8f8) [0x55b80edc28f8] [01:59:20.491] [344] [critical] 5, /usr/bin/infinity(+0x2c1e63) [0x55b80edc5e63] [01:59:20.491] [344] [critical] 6, /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd918c2eac3] [01:59:20.491] [344] [critical] 7, /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fd918cbfa04] [01:59:20.491] [344] [critical] Txn ID: 1, Text: ForceCheckpointTask, Begin TS: 41578, Commit TS: 41579, State: Committed
I also meet the same question ,as below ,I run ragflow with infinity in docker [00:46:08.624] [223] [critical] Txn ID: 29, Text: Select Statement, Begin TS: 85650, Commit TS: 0, State: Started
[00:46:08.624] [223] [critical] Txn ID: 28, Text: Select Statement, Begin TS: 85650, Commit TS: 0, State: Started
[00:46:08.625] [223] [critical] Error: This task should not be scheduled if the source queue is empty@src/executor/operator_state.cpp:50 [00:46:08.625] [223] [critical] 0, /usr/bin/infinity(+0x4e0629) [0x5637ed8e9629] [00:46:08.625] [223] [critical] 1, /usr/bin/infinity(+0x4e1474) [0x5637ed8ea474] [00:46:08.625] [223] [critical] 2, /usr/bin/infinity(+0x13b3e6c) [0x5637ee7bce6c] [00:46:08.625] [223] [critical] 3, /usr/bin/infinity(+0x60ad70) [0x5637eda13d70] [00:46:08.625] [223] [critical] 4, /usr/bin/infinity(+0x2be8f8) [0x5637ed6c78f8] [00:46:08.625] [223] [critical] 5, /usr/bin/infinity(+0x2c1e63) [0x5637ed6cae63] [00:46:08.625] [223] [critical] 6, /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f52e41fcac3] [00:46:08.625] [223] [critical] 7, /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f52e428da04] [00:46:08.625] [223] [critical] Txn ID: 1, Text: ForceCheckpointTask, Begin TS: 85644, Commit TS: 85645, State: Committed Checkpoint: catalog path: catalog/FULL.85643.json max commit ts: 85643 is full checkpoint: 1
ForceCheckpointTask
[00:46:08.625] [223] [critical] Txn ID: 9, Text: Full or delta checkpoint, Begin TS: 85646, Commit TS: 85647, State: Committed Checkpoint: catalog path: catalog/FULL.85645.json max commit ts: 85645 is full checkpoint: 1
[00:46:08.625] [223] [critical] Txn ID: 19, Text: Full or delta checkpoint, Begin TS: 85648, Commit TS: 85649, State: Committed Checkpoint: catalog path: catalog/FULL.85647.json max commit ts: 85647 is full checkpoint: 1
[00:46:08.625] [223] [critical] Txn ID: 29, Text: Select Statement, Begin TS: 85650, Commit TS: 0, State: Started
[00:46:08.625] [223] [critical] Txn ID: 28, Text: Select Statement, Begin TS: 85650, Commit TS: 0, State: Started
[00:46:08.625] [223] [critical] Error: TerminateHandler: Unhandled Exception: This task should not be scheduled if the source queue is empty@src/executor/operator_state.cpp:50 [00:46:08.625] [223] [critical] 0, /usr/bin/infinity(+0x4e0629) [0x5637ed8e9629] [00:46:08.625] [223] [critical] 1, /usr/bin/infinity(+0x22fcf5) [0x5637ed638cf5] [00:46:08.625] [223] [critical] 2, /usr/bin/infinity(+0x1e0d383) [0x5637ef216383] [00:46:08.625] [223] [critical] 3, /usr/bin/infinity(+0x1e0f7e6) [0x5637ef2187e6] [00:46:08.625] [223] [critical] 4, /usr/bin/infinity(+0x1e0f7d0) [0x5637ef2187d0] [00:46:08.625] [223] [critical] 5, /usr/bin/infinity(+0x4e1500) [0x5637ed8ea500] [00:46:08.625] [223] [critical] 6, /usr/bin/infinity(+0x13b3e6c) [0x5637ee7bce6c] [00:46:08.625] [223] [critical] 7, /usr/bin/infinity(+0x60ad70) [0x5637eda13d70] [00:46:08.625] [223] [critical] 8, /usr/bin/infinity(+0x2be8f8) [0x5637ed6c78f8] [00:46:08.626] [223] [critical] 9, /usr/bin/infinity(+0x2c1e63) [0x5637ed6cae63] [00:46:08.626] [223] [critical] 10, /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f52e41fcac3] [00:46:08.626] [223] [critical] 11, /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f52e428da04] [00:46:08.640] [222] [info] Periodic trigger stop ...
我也遇到了同样的问题，如下所示，我在 docker 中以 infinity 运行 ragflow [00：46：08.624] [223] [critical] Txn ID: 29, Text: Select Statement, Begin TS: 85650, Commit TS: 0, State: Started
[00：46：08.624] [223] [关键] Txn ID：28，文本：选择语句，开始 TS：85650，提交 TS：0，状态：已开始
[00：46：08.625] [223] [严重] 错误：如果源队列为空，则不应安排此任务@src/executor/operator_state.cpp:50 [00：46：08.625] [223] [严重] 0，/usr/bin/infinity(+0x4e0629) [0x5637ed8e9629] [00：46：08.625] [223] [严重] 1，/usr/bin/infinity(+0x4e1474) [0x5637ed8ea474] [00：46：08.625] [223] [严重] 2，/usr/bin/infinity(+0x13b3e6c) [0x5637ee7bce6c] [00：46：08.625] [223] [严重] 3，/usr/bin/infinity(+0x60ad70) [0x5637eda13d70] [00：46：08.625] [223] [严重] 4，/usr/bin/infinity(+0x2be8f8) [0x5637ed6c78f8] [00：46：08.625] [223] [严重] 5，/usr/bin/infinity(+0x2c1e63) [0x5637ed6cae63] [00：46：08.625] [223] [严重] 6，/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f52e41fcac3] [00：46：08.625] [223] [严重] 7，/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f52e428da04] [00：46：08.625] [223] [严重] Txn ID：1，文本：ForceCheckpointTask，开始 TS：85644，提交 TS：85645，状态：已提交 检查点： 目录路径：catalog/FULL.85643.json 最大提交 ts：85643 已满 检查点：1
强制检查点任务
[00：46：08.625] [223] [关键] Txn ID：9，文本：完整或增量检查点，开始 TS：85646，提交 TS：85647，状态：已提交 检查点： 目录路径：catalog/FULL.85645.json 最大提交 ts：85645 已满检查点：1
[00：46：08.625] [223] [关键] Txn ID：19，文本：完整或增量检查点，开始 TS：85648，提交 TS：85649，状态：已提交 检查点： 目录路径：catalog/FULL.85647.json 最大提交 ts：85647 已满检查点：1
[00：46：08.625] [223] [关键] Txn ID：29，文本：选择语句，开始 TS：85650，提交 TS：0，状态：已开始
[00：46：08.625] [223] [关键] Txn ID：28，文本：选择语句，开始 TS：85650，提交 TS：0，状态：已开始
[00：46：08.625] [223] [严重] 错误：TerminateHandler：未处理的异常：如果源队列为空，则不应安排此任务@src/executor/operator_state.cpp:50 [00：46：08.625] [223] [严重] 0，/usr/bin/infinity(+0x4e0629) [0x5637ed8e9629] [00：46：08.625] [223] [严重] 1，/usr/bin/infinity(+0x22fcf5) [0x5637ed638cf5] [00：46：08.625] [223] [严重] 2，/usr/bin/infinity(+0x1e0d383) [0x5637ef216383] [00：46：08.625] [223] [严重] 3，/usr/bin/infinity(+0x1e0f7e6) [0x5637ef2187e6] [00：46：08.625] [223] [严重] 4，/usr/bin/infinity(+0x1e0f7d0) [0x5637ef2187d0] [00：46：08.625] [223] [严重] 5，/usr/bin/infinity(+0x4e1500) [0x5637ed8ea500] [00：46：08.625] [223] [严重] 6，/usr/bin/infinity(+0x13b3e6c) [0x5637ee7bce6c] [00：46：08.625] [223] [严重] 7，/usr/bin/infinity(+0x60ad70) [0x5637eda13d70] [00：46：08.625] [223] [严重] 8，/usr/bin/infinity(+0x2be8f8) [0x5637ed6c78f8] [00：46：08.626] [223] [严重] 9，/usr/bin/infinity(+0x2c1e63) [0x5637ed6cae63] [00：46：08.626] [223] [严重] 10，/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f52e41fcac3] [00:46:08.626] [223] [关键] 11，/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f52e428da04] [00:46:08.640] [222] [信息] 定期触发停止...
这个问题解决了么？我也遇到相同的问题
same problem，infinity restart again and again!
2025-04-30 08:59:16 [00:59:16.074] [229] [critical] Error: TerminateHandler: Unhandled Exception: This task should not be scheduled if the source queue is empty@src/executor/operator_state.cpp:50 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 0, /usr/bin/infinity(+0x4e0629) [0x564eb0f70629] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 1, /usr/bin/infinity(+0x22fcf5) [0x564eb0cbfcf5] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 2, /usr/bin/infinity(+0x1e0d383) [0x564eb289d383] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 3, /usr/bin/infinity(+0x1e0f7e6) [0x564eb289f7e6] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 4, /usr/bin/infinity(+0x1e0f7d0) [0x564eb289f7d0] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 5, /usr/bin/infinity(+0x4e1500) [0x564eb0f71500] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 6, /usr/bin/infinity(+0x13b3e6c) [0x564eb1e43e6c] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 7, /usr/bin/infinity(+0x60ad70) [0x564eb109ad70] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 8, /usr/bin/infinity(+0x2be8f8) [0x564eb0d4e8f8] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 9, /usr/bin/infinity(+0x2c1e63) [0x564eb0d51e63] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 10, /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f20f380fac3] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] 11, /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f20f38a1850] 2025-04-30 08:59:16 [00:59:16.074] [229] [critical] Txn ID: 1, Text: ForceCheckpointTask, Begin TS: 53017, Commit TS: 53018, State: Committed 2025-04-30 08:59:16 Checkpoint: 2025-04-30 08:59:16 catalog path: catalog/FULL.53016.json 2025-04-30 08:59:16 max commit ts: 53016 2025-04-30 08:59:16 is full checkpoint: 1
