SketchyVim
SketchyVim copied to clipboard
FelixKratz
Not working in slack
Slack doesn't seem to be working with SketchyVim. Perhaps it's related to it being an Electron app?
I believe I could make it work by setting the "AXManualAccessibility" attribute for electron apps, but I am not sure if that leads to other bugs. I will investigate.
One idea is to start with is making that type of option configurable on a per-app basis, similar to how you've got a blocklist, you could expand to a config file that lets you override certain things. That way you could do this to work-around issues, and it's easy enough to turn back off.
Maybe we can make a list for which apps work well with svim which are not.
A list might be a good idea to better manage expectations. The source code assumes that text accessibility is implemented properly for every app (which it is not, sadly). If the text accessibility was enforced a bit stricter this would work everywhere.
I have an indicator in my SketchyBar, which is green when svim has detected a working text field, and red otherwise:
This is the example script.sh I have bundled.
So I have looked at Slack and I can get it to actually expose some accessibility information at all via:
open -a "/Applications/Slack.app" --args --force-renderer-accessibility
but it does not properly report the caret position anyways. It is always 0, regardless of the text content. So I believe electron apps are just not compatible because of their horrible accessibility integration.
I could do some janky guess work in those cases, as I can set the text, but without proper cursor synchronization this gets a bit too sketchy.
Perhaps we could file an issue within the Electron repo and see if there's any way to fix this there, or if every app on Electron needs to do work to be fixed. If nothing else, they may have suggestions.
In https://github.com/dbalatero/VimMode.spoon I am able to read cursor info from Slack, I just can't set cursor position.
The trick is to set the AXManualAccessibility accessibility flag to true, see here:
https://github.com/dbalatero/VimMode.spoon/blob/master/lib/utils/ax.lua#L37-L41
Electron apps are tricky because even though the Electron framework itself "supports" AX (with the AXManualAccessibility as @dbalatero stated), apps themselves are quite wild with their implementation. some will return proper data, some will not. Slack doesn't. Logseq almost. ClickUp doesn't. etc etc.
btw if i remember well it looks like Slack can't get/set caret position but it can. the issue is that it mixes caret position with line number. so if you write stuff like set caret at position 3 and select 5 characters, what it does is setting your caret at line 3 and juggle with the rest of your data. in my experience, once past the fact that the Electron framework can bring apps everywhere, the rest of the layers are too freestyle to be supported properly.
I can't understand Apple in this regard... All of the apps in the mac app store adhere to very strict guidelines (e.g. no private frameworks like skylight), yet they completely seem to have forgotten about the accessibility. IMO all app store apps should be required to be fully accessible.
A major app not being accessible is like an office building with no ramp or elevator... just horrible and arrogant. Researching the electron accessibility I have come across several posts from people with motor disabilities and apps like Slack are almost unusable for them because they can not select the text properly.
I will not do anything to explicitly support non-accessible apps, all be it to make regular people recognize the arrogance and neglect of the makers of these applications.
well, i call Electron apps "Electron shit apps" and "big fat liar apps" all over kindaVim docs and app 😂️ honestly i don't know the politics about Apple and the App Store but i guess if they'd not allow Electron apps they'd be seen even more as tyrants, monopoly blah blah the usual bullshit. huge backlash from the developers community.
what saddens me is that yeah 1) native apps handle Accessibility because Apple does a lot of good work in this area and it's pretty shit to go around that 2) Electron is for improving the life of developers, not users. using only the knowledge you know, not adapted, to build subpar shits. it's like going to a new country and only speaking your language rather than learning the local one. that's utterly lazy, shitty and arrogant. bad UI. bad UX. but that's the trend. and the market speaks. more users, more market share. more money made 🤷♂️️ personally i like to build useful, simple, apps with great UX. it has to help a user, saving time, friction, or making them smile. native and clean. i don't care much about the rest (but that may explain why i only have 5 paying customers 😂️).
I can't understand Apple in this regard... All of the apps in the mac app store adhere to very strict guidelines (e.g. no private frameworks like skylight), yet they completely seem to have forgotten about the accessibility. IMO all app store apps should be required to be fully accessible.
I absolutely feel this. What's so odd is that plenty of native Mac apps seem to implement all the APIs just fine. When I filed an issue with Electron to fully implement all the accessibility APIs, they shot it down on grounds that it would get all the apps banned from the App Store–which, if they were dealing with tons of end user bug reports, I can understand why that might be their policy. It just makes no sense that the App Store would slap down Electron apps for using "private APIs" when the intent of those same APIs is to provide an accessible experience. I don't think anyone would disagree that that is a good software goal.
Researching the electron accessibility I have come across several posts from people with motor disabilities and apps like Slack are almost unusable for them because they can not select the text properly.
That's honestly pretty painful to hear.
I will not do anything to explicitly support non-accessible apps, all be it to make regular people recognize the arrogance and neglect of the makers of these applications.
I tried to implement different levels of support on my library (reproducing simple Vim motions for apps that don't support accessibility, and doing fancier stuff for better native apps), and I feel like I ended up creating a split, confusing experience that I'm not really that happy with. Honestly if the accessibility API hadn't gotten in the way, I would have poured hundreds of extra hours of enthusiasm into creating the best Vim-everywhere user experience I possibly could, but as a result I feel really stalled out on the project.
I wish we could somehow move the mountain that is Apple + Electron and try to get better support baked in here. While it's tempting to throw up my hands and say "screw all the Electron apps," there are so many in the wild that it makes it pretty disappointing.
I experimented a bit with a deeper system integration and was able to circumvent the accessibility limitations by gaining direct text access via the systems text input server, but stumbled over some other obstacles with the new approach. It is definitely possible to get this tool working in any text field, regardless of accessibility integration, but I will have to fiddle with it some more.
@FelixKratz Wow can you share more about where I can read about this text input server?
Also can you share the obstacles you've encountered? I might be able to help.
I experimented a bit with a deeper system integration and was able to circumvent the accessibility limitations by gaining direct text access via the systems text input server, but stumbled over some other obstacles with the new approach. It is definitely possible to get this tool working in any text field, regardless of accessibility integration, but I will have to fiddle with it some more.
WOW. that would be an amazing feat.
technically i'm very curious. that means you'd bypass the AX. can you still set caret location etc?
also not related to this, but i'm wondering how you gonna handle Screen Moves in SketchyVim. like zz, H etc. using a Vim backend is fantastic for Text Moves, but for Screen Moves personally the only way i found it possible was to use macOS APIs first rather than Vim (buffer) first (which meant rewrite all those moves).
I will share a bit about my idea and why I believe it should be possible: The text input is more than just keys being send to an application, there are numerous features that the system controls. For example: the system manages the dictionary, spelling correction, character conversion to e.g. chinese character, etc. The system obviously needs to have access to those text fields and can read, write, set caret etc. in those text fields. Any application that wants to be compliant with the apple input server and the features above needs to properly comply with these system apis (and that would be all relevant applications where you would want vim moves).
Now for my proof of concept: Getting access to those privileges can be achieved in numerous ways and the simplest is to register a new input method with the system e.g. via apples Input Method Kit. This gives read write access to text and selected and marked ranges already, as well as an event tap for the key events, theoretically providing everything needed. I have implemented this and it indeed works fine for reading and setting the text in all applications, also Slack etc. because now we are at a totally different privilege level and do not need the accessibility api at all to do those things. However, the input method kit is not well equipped to properly set the marked range style and I struggled to make it look nice. It might be possible to use the privileges acquired to use some lower level api calls for that, but I'll have to investigate more.
The second approach that would be even more powerful would include disabling system integrity protection and loading a payload directly into the text input server. This would have some benefits, but also the massive regression of having to update the payload for every major OS release and patching the binary for x86 and arm64 separately, also this would include a fair bit of reverse engineering work.
The Input Method Kit sounds very promising! I always wanted a stable API to work with, because then one could actually boot a headless neovim in RPC mode and proxy text/cursor back and forth, and inherit all the already implemented Vim motions for free.
The second approach that would be even more powerful would include disabling system integrity protection and loading a payload directly into the text input server. This would have some benefits, but also the massive regression of having to update the payload for every major OS release and patching the binary for x86 and arm64 separately, also this would include a fair bit of reverse engineering work.
For me, I would never be able to run this at work due to our IT security controls, and probably many users might be in the same boat here as well (or uncomfortable with disabling SIP for personal reasons). It's certainly worth an exploration to see what is possible, though!
Now for my proof of concept: Getting access to those privileges can be achieved in numerous ways and the simplest is to register a new input method with the system e.g. via apples Input Method Kit. This gives read write access to text and selected and marked ranges already, as well as an event tap for the key events, theoretically providing everything needed. I have implemented this and it indeed works fine for reading and setting the text in all applications, also Slack etc. because now we are at a totally different privilege level and do not need the accessibility api at all to do those things.
smart. very nice.
The second approach that would be even more powerful would include disabling system integrity protection and loading a payload directly into the text input server. This would have some benefits, but also the massive regression of having to update the payload for every major OS release and patching the binary for x86 and arm64 separately, also this would include a fair bit of reverse engineering work.
yeah that one for me is/was out of question. disabling SIP would be a no no, both as a user and the dev.
So today I started to use Mailspring, an Electron app, and I wanted svim to work in this app. So I decided to set the kAXManualAccessibility flag for apps if svim is compiled with '-DMANUAL_AX'.
