core icon indicating copy to clipboard operation
core copied to clipboard

Published 20 hours ago • verified publisher icon cheshire-cat-ai

add multilingual support

Open pieroit opened this issue 2 years ago • 1 comments

Default prompts in chains and agents are in english.

  • OPTION 1: localization There should be a language detector to classify user input and to load default prompts in the appropriate language. (localized content should be organized in a similar manner as in CMSs like WordPress)

  • OPTION 2: wrapper Run the language detector, translate user input in english using a translation model, run the pipelines/agents/chains in english and then translate back the final response back to user language.

Let's go with option 2 as llms are weak in languages other than english

pieroit avatar Mar 01 '23 09:03 pieroit

I developed a multilingual FAQ Search a year ago using the DeepL API and xlm model.

nickprock avatar Apr 08 '23 05:04 nickprock

Other option, we can use XLM. I tested with openaAI and if a give the input in italian it responds in italian, for HF we can force a cross language model (xlm) and multilingual embeddings

nickprock avatar Apr 28 '23 09:04 nickprock

Hooks to change prompt language are now exposed

pieroit avatar May 04 '23 07:05 pieroit