cursorless
cursorless copied to clipboard
Add user setting for tokeniser
The user should be able to override the values of https://github.com/cursorless-dev/cursorless/blob/2098df631cdb9fb597b0f8eb90f5bd210a47186f/src/core/tokenizer.types.ts#L3-L9 in their settings. This would be similar to the way that languages override them, eg https://github.com/cursorless-dev/cursorless/blob/4906e2b24845a89557851ba0cca8136d12b8d32e/src/core/languageTokenizers.ts#L9
Note that users will be able to use language-specific overrides just via VSCode language settings
This setting could be used to accomplish https://github.com/cursorless-dev/cursorless/pull/986 on a per-user basis
I wonder if it would be simpler to just expose a wordSeparators
setting that by default just includes _
. Then they could just remove or add to that list without needing to mess with the full regex
cc/ @phillco @AndreasArvidsson
That's probably the easiest way of doing it.
That would work.
How would this setting interact with language specific preference in Cursorless? For example if I have this setting globally set to just _
would that take precedents over the preference for css? Because I actually found myself not liking the tokenizer in css and would really like to have the same one in every language.
Or should we just remove the language specific regex and totally rely on this setting alone?