cursorless icon indicating copy to clipboard operation
cursorless copied to clipboard

Add user setting for tokeniser

Open pokey opened this issue 2 years ago • 3 comments

The user should be able to override the values of https://github.com/cursorless-dev/cursorless/blob/2098df631cdb9fb597b0f8eb90f5bd210a47186f/src/core/tokenizer.types.ts#L3-L9 in their settings. This would be similar to the way that languages override them, eg https://github.com/cursorless-dev/cursorless/blob/4906e2b24845a89557851ba0cca8136d12b8d32e/src/core/languageTokenizers.ts#L9

Note that users will be able to use language-specific overrides just via VSCode language settings

This setting could be used to accomplish https://github.com/cursorless-dev/cursorless/pull/986 on a per-user basis

pokey avatar Oct 02 '22 17:10 pokey

I wonder if it would be simpler to just expose a wordSeparators setting that by default just includes _. Then they could just remove or add to that list without needing to mess with the full regex

cc/ @phillco @AndreasArvidsson

pokey avatar Oct 03 '22 08:10 pokey

That's probably the easiest way of doing it.

AndreasArvidsson avatar Oct 03 '22 08:10 AndreasArvidsson

That would work.

phillco avatar Oct 03 '22 17:10 phillco

How would this setting interact with language specific preference in Cursorless? For example if I have this setting globally set to just _ would that take precedents over the preference for css? Because I actually found myself not liking the tokenizer in css and would really like to have the same one in every language.

Or should we just remove the language specific regex and totally rely on this setting alone?

AndreasArvidsson avatar Oct 23 '22 09:10 AndreasArvidsson