Dify misidentified MARKDOWN file segmentation

[yu154]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Parent segmentation：### content1 Subclass segmentation：#### content2

✔️ Expected Behavior

content1 （Parent ） content2（Subclass）

❌ Actual Behavior

content1 （Parent ） #content2（Parent ）

[dosubot[bot]]

Hello, @yu154! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's work together to resolve the issue you're experiencing.

The issue with Dify misidentifying the segmentation of a MARKDOWN file likely stems from an incorrect JSON payload structure. Specifically, the 'parent_mode' and 'subchunk_segmentation' parameters might be misplaced. They should be placed under the 'rules' object, not directly under 'process_rule'. Here's the correct structure:

{
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "segmentation": {
                "separator": "whatever",
                "max_tokens": 2000
            },
            "parent_mode": "paragraph",
            "subchunk_segmentation": {
                "separator": "***",
                "max_tokens": 500
            }
        }
    }
}

Ensure that the segmentation rules are set according to the document format and context requirements. Proper configuration is crucial for accurate content classification and retrieval ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[DavideDelbianco]

I confirm the same behavior; I've also tried with regular expressions. Considering that most tools tend to convert scraped / extract content to markdown, this is an issue. Using \n as suggested in the UI to separate chunks is a bad practice in RAG, because you will end up with very little information in each chunk vector and considering that retrieval in Dify is based only on a max TopK of 10 this ends up generating useless response from the AI.

[rowstop]

I encountered the same problem, I haven't read the source code, so I'm not sure what caused it. But I solved it by customizing the segmentation symbol: <!-- p -->and <!-- c -->

[rowstop]

😅How can we solve it?

[dosubot[bot]]

Hi, @yu154. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

• A bug in Dify version 0.15.3 causes incorrect Markdown file segmentation in a Docker environment.
• "content2" is mistakenly identified as a parent instead of a subclass under "content1".
• I suggested the issue might be due to an incorrect JSON payload structure and provided a corrected example.
• @DavideDelbianco confirmed the issue, noting that \n for chunk separation is ineffective.
• @rowstop shared a workaround by customizing segmentation symbols, with ongoing community discussions for solutions.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Dify repository. If so, you can keep the discussion open by commenting on the issue.
• Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!