Knowledge API: Create a Document from a File URL

[XU-Nuo]

Self Checks

• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• [x] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I am building an ocr pipeline that can create/update a knowledge base, which stores the ocr results to s3 and uses the knowledge /datasets/{dataset_id}/document/create-by-file API to update the knowledge base.

Unlike many other APIs like openai that allow passing a file url as the file, the current create-by-file API asks for a path from the system disk, meaning that I need to download the file into a temporary folder first from s3 and then call the API using the local path which adds extra time and bandwidth costs.

2. Additional context or comments

document/create-* API should support more formats, inlcuding from text, local file system, url, and possibly base64 bianry for non-plaintext file like docx and pdf, etc.

3. Can you help us with this feature?

• [ ] I am interested in contributing to this feature.

[dosubot[bot]]

Hi, @XU-Nuo. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• Enhancement proposed for the Knowledge API to support document creation via file URL.
• Suggested improvements include support for multiple formats: text, URLs, and base64 binary.
• No further comments or activities have been made on this issue.

Next Steps:

• Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!