Bedrock cohere fix max length 2048

[kerlion]

Description

It includes 2 changes:

1. For Cohere, count the tokens as text length to avoid 2048 length limitation..
2. For others, count the tokens with tiktoken tokennizer, uses gpt-4 instead of gpt2.

Fixes #3942

Type of Change

Please delete options that are not relevant.

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update, included: Dify Document
• [x] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• [ ] Dependency upgrade

How Has This Been Tested?

Test with text file with consecutive punctuation marks or long English words. It will failed with expected maxLength: 2048 before. Now it is fixed by counting the tokens as chars.

• [ ] TODO

Suggested Checklist:

• [x] I have performed a self-review of my own code
• [x] have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [ ] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• [ ] optional I have made corresponding changes to the documentation
• [ ] optional I have added tests that prove my fix is effective or that my feature works
• [ ] optional New and existing unit tests pass locally with my changes

[crazywoola]

@kerlion Please resolve the conversation above, we will merge it later.

[crazywoola]

Hello please fix the conflicts :) dm on wechat if you finished this -> crazyphage

[crazywoola]

Close for now. You can resolve those conflicts and open a new one.