allow to config max segmentation tokens length for RAG document using environment variable

[rainchen]

Description

allow to config max segmentation tokens length for RAG document using environment variable

Fixes # (issue)

Fix #2438 #3290 also #4071

Type of Change

Please delete options that are not relevant.

• [x] This change requires a documentation update, included: Dify Document
• [x] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement

How Has This Been Tested?

manually tested

Suggested Checklist:

• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• [ ] optional I have made corresponding changes to the documentation
• [ ] optional I have added tests that prove my fix is effective or that my feature works
• [ ] optional New and existing unit tests pass locally with my changes

[JohnJyong]

pls add the default value for the new variable in the file : config.py , thanks @rainchen

[rainchen]

pls add the default value for the new variable in the file : config.py , thanks @rainchen

@JohnJyong refactored, please review again

[JohnJyong]

LGTM

[rennokki]

@rainchen AMAZING PR! I was wondering if something like this is possible because I was having some texts that were > 1k and they got broke down.

[daigo38]

I have set the docker-compose.yaml environment variable on self-host and restarted the container, but the changes are not applied. I am building with the “langgenius/dify-api:0.6.8” image, is this feature already enabled?