[Bug] Search for `type` in dialog.type() leads to no result in EN locale

[mxschmitt]

Hello,

currently when using the English locale and searching for type in the text dialog.type() it leads to no search results. When using zh as a locale it works. Also when searching for the full match (dialog.type()) it works.

Seems like we need to use the tokenizer.ts for English as well.

[weareoutman]

Um, this is probably caused by the default separator of the search library under the hood named lunr.js, see here, which uses /[\s\-]+/ as the default separator. It's reasonable to not treat . as a separator for normal text content.

When using zh as a locale, it overrides the default tokenizer which ignores the default separator. As a simple solution maybe we can expose an option to allow users to override the default separator (for non-zh locales).

[code-masala]

@mxschmitt how did you solve this problem.I have the regional language these language doesn't belong to the set of language which is provide by docusaurs.When I try it give Error: Cannot find module 'lunr-languages/lunr.bn' where "bn" is the regional language. For all language in build ,search-index.json is generated perfactly but in search bar only supported default locale which is the english.