langgenius
Unable to add in batch in the knowledge base
Self Checks
- [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
- [x] I have searched for existing issues search for existing issues, including closed ones.
- [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
- [x] Please do not modify this template :) and fill in all the required fields.
Dify version
0.15.3
Cloud or Self Hosted
Cloud
Steps to reproduce
Problem Description
I'm using a custom text chunking method to complete text chunking and then save it as a csv file in the batch add format, but it reports an error when importing in version 0.15.3 of Dify. It is known to be able to import successfully in version 0.11.2 using the same file. Also, as far as I know, it is no longer possible to successfully batch-add since at least version 0.15.1. Is it possible to fix this problem?
Error: Run batch failed
✔️ Expected Behavior
Import completed
❌ Actual Behavior
Run batch failed
@KamenC Hey there! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's squash those bugs together! 🦾
I found a similar closed issue that might be relevant to your problem: Knowledge Base Document Q&A Format Batch Add Segment Failure. This issue was resolved by a pull request that fixed the pandas indexing method for knowledge base imports [1].
In Dify version 0.15.3, specific error messages are generated when a batch import fails. These include errors related to file structure mismatches, empty files, and invalid or excessively long values in specific rows [2].
The CSV file used for batch imports must have a specific structure with columns for "pergunta" (Question), "resposta" (Answer), and "conteúdo" (Content) [3][4].
Given that your issue involves a custom text chunking method, it might be worth checking if the CSV file structure aligns with these requirements. Additionally, reviewing the changes made in the pull request that fixed the previous issue might provide insights into resolving your current problem.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other | Bug Report
![dosubot[bot] avatar](https://avatars.githubusercontent.com/in/324583?v=4 "Published by a pub.dev verified publisher"){kind=small}
Can you provide some example files for us to test it out.您能否提供一些示例文件供我们测试。
I'm sorry, the file I'm using is confidential. But I get the same problem using the batch add template file provided by Dify. Here is the template file provided by Dify for you to try, thanks a lot!
Seems not able to reproduce it. Can you provide more details that you are trying to achieve here.
Seems not able to reproduce it. Can you provide more details that you are trying to achieve here.
No, not uploading it here, where it would change my original text chunking strategy. I am uploading a test file containing a small amount of data for creating the knowledge base first.
Then, enter that file and click the ‘Batch add’ button to select the csv file to upload:
Hi, @KamenC. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.
Issue Summary:
- Bug reported in Dify version 0.15.3 with batch adding data using custom text chunking.
- Issue not present in version 0.11.2.
- I suggested checking CSV structure and reviewing a related resolved issue.
- Crazywoola requested example files; KamenC provided a template, but the issue couldn't be reproduced.
- Fniu confirmed experiencing the same problem, indicating a potential widespread issue.
Next Steps:
- Please confirm if this issue is still relevant to the latest version of Dify. If so, you can keep the discussion open by commenting here.
- If there is no further activity, this issue will be automatically closed in 15 days.
Thank you for your understanding and contribution!