[CON-274] Take into account maximum token count for embeddings provider batches

[sestinj]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:*
- Continue:*
- IDE:*

Description

Example error here: https://discord.com/channels/1108621136150929458/1289672061568028722

We have a maxBatchSize, but not a maxBatchTokens, which causes us to go over limits for some APIs (e.g. Mistral limits are 128 and 16,384, so with chunk size 512 * 128 = 65,536, we exceed the limit by a bit)

To reproduce

No response

Log output

No response

_CON-274