dify File indexing failed and report an error

Self Checks

[x] This is only for bug report, if you would like to ask a question, please head to Discussions.
[x] I have searched for existing issues search for existing issues, including closed ones.
[x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When uploading files to the knowledge base for vectorization, file indexing failed and report an error:PluginlnvokeError: {"args":{},"error type":"RuntimeError","message":"can't start new thread

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Mar 02 '25 07:03 starlitshan

Hello @starlitshan! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's tackle this together!

The "PluginInvokeError" with the message "can't start new thread" is likely due to insufficient system resources, even if your system seems to have enough RAM and a high threads-max limit. This issue has been observed in Dify when using the worker service with plugins like Ollama for embeddings. To resolve this, you may need to significantly increase your system resources, such as upgrading your host memory to 24GB and increasing the threads-max limit to a much higher value, like 752215 ^[1].

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

Mar 02 '25 07:03 dosubot[bot]

I also encountered the same error

Mar 02 '25 10:03 0Knot

同样的问题，特别是向量化较大信息量文件的时候（字符数 > 100k）

Mar 03 '25 07:03 Ryaningli

重启容器就好了，但这不是解决办法呀

Mar 04 '25 01:03 SoftBunny

这个问题同时也会影响到知识检索，会超时，日志里报的错同样是can't start new thread，确实重启容器会好。

Mar 04 '25 02:03 Ryaningli

752215

@crazywoola the cpu and ram resources are all OK, but the same errer happen~ I have set the threads-max limit to 752215

even there are no calling record from the embedding service log.

Mar 04 '25 14:03 meitianjinbu

fix https://github.com/langgenius/dify/issues/14626
fix https://github.com/langgenius/dify/issues/14670
fix https://github.com/langgenius/dify/issues/14659

From

plugin_daemon:
  image: langgenius/dify-plugin-daemon:0.0.3-local

To

plugin_daemon:
  image: langgenius/dify-plugin-daemon:942d694206a1972ae0addd2286a021b62390a8b6-local

Mar 05 '25 05:03 crazywoola