[Feature request] multiple click mode

[albert-ying]

In many scenarios, people will need to perform multiple clicks on the same page (e.g., check the boxes in a form). In the current Vimac, if I want to click 3 buttons on a page, I will need to do: ctrl-space hint1 ctrl-space hint2 ctrl-space hint3.

This is cumbersome and slow (lag time x 3).

Requested feature: Add a muti-click mode that allows the user to click at multiple hints, instead of removing hints from screen immediately after the first click.

So in the previous example, if I want to click 3 buttons on a page, I will only need to do: ctrl-m hint1 hint2 hint3 esc.

[dexterleng]

I agree there is a need to perform multiple clicks that is easier than ctrl-space hint1 ctrl-space hint2 ctrl-space hint3..

My hypothesis is that the cost of doing hand stretching to hitting the shortcut to activate hint-mode is higher than simply using the mouse.

I don't think Vimium users are requesting the "multi-click" feature you suggested because hitting F is so easy and cheap - it is right below their index finger.

IMO, There are two approaches to solve this problem:

1. Optimizing the way hint-mode can be triggered

I think the goal is to make it as easy as Vimium -> hit the F key to activate hint-mode.

My first attempt to achieve this was with key-sequence activation (e.g. type fd to activate hint-mode). @albert-ying have you given this a try?.

Here are some ideas I have

Workflow mode

User hits a shortcut to activate Workflow mode.

This mode acts exactly like Vimium:

• f: activates hint-mode, after clicking it returns back to workflow mode
• hjkl: scrolls within workflow mode
• i: insert mode for typing into main document.
• esc: exit workflow mode

Workflow mode will automatically disable itself when it detects keyboard focus on the main application (just like Vimium), and automatically reactivate when keyboard focus is lost or when a new window is the frontmost.

Supporting simpler shortcuts

Perhaps single or double tapping of modifier keys can be supported to make it easier to activate modes without using keyboard software like Karabiner. (e.g. tap Option to enable hint-mode)

2. Continuous hint-mode (your idea)

After a click is performed, hint-mode is reactivated. A challenge here is that there are certain animations, loading, or screen changes that take time to complete. Perhaps reactivating hint-mode instantly isn't the best solution. Maybe the user can specify a "reactivation delay". Or have a key (e.g. R) for refreshing the hints while hint-mode is activated.

[dexterleng]

This problem of reducing the cost of activating Vimac has many solutions (including the Workflow mode idea I've suggested). I'd love to hear everyone's thoughts and suggestions.

[mmroczka]

Love the thoughts here! Here's some input from a daily Vimac user.

I think perhaps you're solving the wrong problem. Currently, hint mode is not particularly difficult to activate, it's just difficult to reactivate quickly. Right now, I've assigned my hint mode shortcut to control + shift + command + option + f. It sounds difficult, but I've created a hyper key that remaps capslock into control + shift + command + option using karabiner elements so I can effectively trigger Vimac with capslock + f and it's very easy since it's both quick to use still and familiar from a vim point of view. With that context...

Issues

There are two issues where Vimac is difficult to use.

Issue 1:

I need to click the same button multiple times (like a counter in a GUI - see screenshot)

So to increment the counter on this GUI up by three units I'd need to...

• capslock + f to activate Vimac
• type two characters associated with the button I want to press
• capslock + f to activate Vimac
• type two more characters associated with the button I want to press
• capslock + f to activate Vimac
• type two characters associated with the button I want to press

That's effectively 12 keystrokes which is too high of a cost and I revert to the mouse.

Issue 2:

I'm in a dialog of some sort and need to click multiple buttons quickly (like a counter in a GUI)

Suggestion 1 would be a cool additional way to trigger Vimac and depending on how it's implemented and the speed at which it could be switched on/off I definitely think there's a possibility I'd switch to it, but it's a bit of a non-issue for me.

Suggestion 2 would resolve issue 2, but still would be tedious for issue 1 (having to wait for the hints to update and click 2 random keystrokes every time I still want to click the same place would be a minimum of 7 keystrokes still).

Additional/Alternate Suggestion(s)

I wasn't sure if I should put this here or move it out separately into a feature request so feel free to tell me to move it!

Suggestion 3a (likely easy to do with minimal dev time) - Move the mouse to the clicked location and leave it there

There are other ways to trigger a mouse click through karabiner or hammerspoon or even baked into some keyboards. A simple fix for issue 1 could be a toggle setting in Vimac that moves the mouse after it clicks an element rather than leaving it where it currently is. Then outside of Vimac if a user needs to trigger that same mouse click multiple times they could do so with other methods.

Suggestion 3b (preferred by likely more dev time) - Implement repeat key [.]

Vim allows us to repeat the previous action as many times as we'd like by using the repeat key . (period). For a user, we could... a) override the default behavior of the period key for 0.x seconds after clicking on something with Vimac. This effectively breaks the period key for a split-second but gives us the ability to click multiple times rapidly. We could set it to an adjustable timeout range (e.g. 0.0x - 5 seconds). OR b) provide a default hotkey that repeat-clicks the last (x,y) point that Vimac clicked (just like Vimac hint mode) and let us override it. If you did that, I'd probably set the shortcut to capslock + . to have the same end effect. a Suggestion 3b with the override is most appealing to me personally.

[blackketter]

Here's a suggestion: Specify a key that, once tapped, can be used to "lock" hint mode for repeated operations.

Then if you tap that key anytime while hint mode is displayed, hint mode does not cancel until you tap that key again (or tap esc).

For example, if that key was set to "r" (for repeat), you could tap:

1. ctrl-f (bring up hint mode)
2. r (lock hint mode on aka "repeat")
3. aa (for a hint)
4. ab (for another hint)
5. r or esc or ctrl-f (unlock hint mode)

You may need to refresh the hints between hint invocations in case the layout changes.

You might also have other keys that lock on other features:

Tap d for double-click Tap option, command, ctrl, & shift to hold down those keys.

So tapping:

ctrl-f r d shift aa ab ac esc

would execute a shift double-click on the hints AA, AB and AC.

To make this obvious, you might want to change the color of the hints or put up an indicator showing what operations are turned on. You might show something like this if repeat, double and shift mode were on:

[albert-ying]

The vb4c (former cVim, a chrome addon similar to Vimium but with more function) has a "multiple click" mode as I described, which can be triggered by keystrokes mf. The screen change should not be a problem, as people tend to only use this mode when they know where to click next, which means that the buttons are already on the screen (similar to mmroczka's counter example).

Workflow mode also sounds great to me!

[dexterleng]

Really awesome ideas in this thread!

I think picking or creating the correct workflow for performing mice actions in VImac will require an understanding of what Vimac users aim to accomplish.

I have summarized a list of actions a user aims to complete, and how they can be accomplished with the cursor, and with Vimac. Orange indicates a non-ideal workflow, and red indicates not supported.

Do let me know if there is anything here that is missing.

[blackketter]

Quick question: in your table above you say "Not all text fields are found..." and "e.g. Info.plist". I don't understand this issue. Can you give another example?

[dexterleng]

"Not all text fields are found..."

I noticed some text fields in web browsers not being hinted. I think they show up as divs or text elements rather than text fields.

"Not Focusable by clicking in the center of the text field"

This is really specific, but I noticed text fields in plist cannot be focused by clicking in the center of the text field "LP", but is only focusable when the text content (HintCharacters) is clicked. Not sure how common this is. In such cases firing an Accessibility "Focus" event or clicking on the text content location might be better than simply clicking at the location.

[blackketter]

Got it, I've seen both of those issues. :)

For the second one, Mail.app looks similarly strange (though it does work):

For these wide items, I wonder if it would be safe to move the hint to the left, where it might be more obvious and/or useful.

[blackketter]

I wonder if Vimac could have some operations that move the cursor without necessarily clicking.

Expanding on my suggestion on locking modifiers, maybe there's a modifier that is move to location, but not click.

As an example for hovering, maybe the m key is for move only. I could hover over an item by typing

ctrl-f - trigger Vimac to display hints m - move only AB - the hint that I want to move to

The cursor would be moved to that location and which would cause a hover item to appear.

Then I could use another Vimac invocation to click on the newly revealed item.

Generally, I think that keyboard controlled mouse movement would be a good addition to Vimac. Both with hints like I just described as well as just movement, similar to the scroll mode that Vimac already has. This would help address the issue of text fields that can't be found/focusable, as well as drag-n-drop and text selection.

[dexterleng]

I've given some thought to the ideas presented here and weighed the benefits/disadvantages.

Here are some important attributes I kept in mind while designing the new hint-mode workflow proposal, which can help explain why it is the way it is:

• Support clicking the same element multiple times easily
• Avoid relying on the modifier keys (see #204)
• Easy to support additional types of "actions" (e.g. text field focus) in the future

Something that it does not support though is @albert-ying's need for an easier way to click a different button right after another. I will have to do some future thinking, but I think this problem can be solved by optimizing how easily "Activation" can be done.

As always, feedback is appreciated.

Proposed New Hint Mode Workflow: Count and Action as a Prefix

Workflow

1. Activation
2. Type Count (optional, default is 1)
3. Type Action key
   • M - Move
   • C - Left Click (Ideally optional)
   • R - Right Click
4. Type Hint Text
5. A click action is perform count times

Example

• Hint Text is "AF"
• To click 10 times: "10AF"
• To right click once: "RAF"

Challenges

• How to show the user the currently typed text?
• Avoid the user having to hit Enter
• Hint action characters / hint text characters ambiguity
  • See Disadvantages

Advantages

• Vim-like
• Simple to implement
• Can easily expand the list of "action keys"
  • F - Text Field Focusing
  • C - Copy text to clipboard
  • S - Select text

Disadvantages

• The default action most users want to perform is single left-click
• Having to type "C" out is slower than the current workflow
• We can avoid this by:
  • Making action key optional and if not provided, the action is left-click
  • To avoid ambiguity, Hint characters cannot start with an action key (e.g. "ma" is not a valid hint-text)
  • Simple solution is to make sure Set of Action Keys do not intersect with Set of Hint Characters

[blackketter]

Nice! Some thoughts:

• What's the timing for multiple clicks? If I typed a 2 would that be a double click?
• Would it be possible to also support tapping of the modifier keys to have them held down during the clicks? It seems more intuitive for me to tap Option to hold down the Option key during the subsequent click(s)? I definitely don't like holding down the modifiers continuously while typing. And they would not use up action keys.
• Re: "How to show the user the currently typed text?" I think a little popup/HUD window with the current configuration could be very helpful. (Like I illustrated above)
• Re: "Avoid the user having to hit Enter" - Not sure what this means
• I don't think that the same key should be used for both hints and actions, if possible.
• If you make the click actions toggle on-off, you could have the left click on by default. tapping that key would turn it off (i.e. making it the same as move!). And not tapping it at all would be the default left-click action.
• I think there should be an action key which locks Vimac activation on.
• Cursor keys pressed while Vimac is on would move the cursor around.

Suggested default key mapping:

ctrl-f - Activate Vimac

L - Toggle left click action (default on) R - Toggle right click action 1 - 9 - Multiple click count M - Multiple clicks (don't deactivate after a click)

shift option cmd ctrl - Toggle modifier keys

A B C etc... - Hint keys

up down left right - Cursor keys move mouse pointer space - Click mouse at current cursor location

ctrl-f or esc - Deactivate Vimac

Some examples: ctrl-f A B - Left click on AB hint ctrl-f 2 A B - Double-click on AB hint ctrl-f shift A B - Shift left click on AB hint ctrl-f R A B - Right click on AB hint ctrl-f M A B C D esc - Click on AB hint, then CD hint ctrl-f L A B esc - Move cursor (but do not click) to AB hint ctrl-f left left up up space - Move the cursor left and up and then click ctrl-f L A B left left left space - Move cursor to AB, move it to the left by 3 increments then click ctrl-f up up down down left right left right B A start - triggers Easter egg :)

[dexterleng]

What's the timing for multiple clicks?

If by timing you mean the interval between clicks, I think I'll have to measure my own mouse clicking speed and follow that.

If I typed a 2 would that be a double click?

Yep

Would it be possible to also support tapping of the modifier keys to have them held down during the clicks?

Great idea!

Re: "Avoid the user having to hit Enter" - Not sure what this means

If the "end" of a typed command cannot be determined (for example if both ab and abb are valid command) then to address that we need either some kind of timer or have the user manual hit Enter. I would like to avoid this.

I don't think that the same key should be used for both hints and actions, if possible.

Right, perhaps the default configuration should be home row keys for hints, and others for actions.

If you make the click actions toggle on-off, you could have the left click on by default. tapping that key would turn it off (i.e. making it the same as move!). And not tapping it at all would be the default left-click action.

I really like this idea of thinking in terms of toggles (with sensible and customizable defaults) rather than commands. You can even do toggling in the middle of typing a hint-text when you change your mind! This would only work if Set of Action Keys do not intersect with Set of Hint Characters.

[dexterleng]

Cursor keys pressed while Vimac is on would move the cursor around.

Sounds good. It's a good idea to have a fallback if an application is not accessible. Because vimac is supposed to replace the mice, having a fallback allows them to continue trying out hint mode and building a habit when they need to perform a mouse operation.

[blackketter]

If by timing you mean the interval between clicks, I think I'll have to measure my own mouse clicking speed and follow that.

You may be able to get the system double-click speed as the user set in preferences via https://developer.apple.com/documentation/appkit/nsevent/1528384-doubleclickinterval

[blackketter]

Re: cursor keys - If there are enough keys left for hints, maybe H J K L could also be used for cursor movement. This feels very vi-like and is consistent with scroll mode. (This would require the left toggle key to be moved to another key.)

[michaelmroczka]

I love thread! :D

Oh more thought to throw in there. The issue with most "mouse modes" on keyboards (programmed or built into the hardware) is that it takes a while to get across a screen to the location you want to click. Some mouse modes implement an "accelerate/decelerate" option, but that's really clunky and never has worked well for me.

Quadrant jumping

I like @blackketter's idea for cursor movement, but don't want to just move my cursor for the reasons above, I want to traverse the distance of my large-ish monitor quickly. What if H J K L moved the mouse just as suggested, but a modifier could have us jump by quadrants? Imagine the screen broken up into a NxM grid (NxM could be a setting or infered based on your resolution/monitor size), and if you held down a modifier like control you could move the mouse by quadrant. This could quickly let you get across the screen without having to deal with mouse acceleration/deceleration.

[dexterleng]

There are two different ways I've seen existing quadrant jumping implemented:

quadrant jumping 1: HJKL as directions that cut the screen in half (e.g. keynav, https://github.com/h2ero/XEasyMotion) quadrant jumping 2. A "hint" assigned to each quadrant (e.g. H for top-left, J for bottom-left...). I think Voice Control has that.

Cursor Movement

• Advantages:
  • Low learning curve, it acts very similarly to how users are used to operating the mice
• Disadvantages:
  • Difficult to move the cursor across large or multiple displays

Quadrant Jumping 1

• Advantages
  • No. of key presses scales logarithmically with screen area
  • If mastered I think this is probably the fastest method to move the cursor
• Disadvantages
  • Probably the hardest to learn

Quadrant Jumping 2

• Advantages
  • No. of key presses scales logarithmically with screen area
  • Concept of dynamically generated "hints", because each screen chunk is assigned a hint text
• Disadvantages
  • Have to visually parse the hints several times if the hints are dynamic rather than static (e.g. H is always for top-left).

[dexterleng]

Ideally Vimac should have custom bindings to custom actions so users can pick their ideal form of navigation, and a sensible default is provided for them.

Perhaps the community can even write extensions that integrate with Vimac!

[blackketter]

This gives me an idea that's more Vimac-like. Why use just 4 keys to move the cursor and not use all of them?

If Vimac had a cursor navigation mode, it could display a grid of hints that cover the whole screen, say 100 of them. Trigger this with, say, the G key. It would then put up hints all over the screen:

AA AB AC AD... BA BB BC BD... CA CB CC CD... ...

Typing BB would jump the cursor to that hint. Then within the area of that grid cell, it could show a smaller grid of hints and then the user could narrow it down from there. And this could continue until the grid cells are too small (probably after just a couple of iterations.). At that point the HJKL keys could be used to fine tune the position of the cursor. Tap the space bar to click.

Alternatively, Vimac's regular hint mode could place additional hints in empty areas all around the screen as quick jump points, though this would be visually very cluttered.

[michaelmroczka]

There are two different ways I've seen existing quadrant jumping implemented:

quadrant jumping 1: HJKL as directions that cut the screen in half (e.g. keynav, https://github.com/h2ero/XEasyMotion) quadrant jumping 2. A "hint" assigned to each quadrant (e.g. H for top-left, J for bottom-left...). I think Voice Control has that.

Cursor Movement

• Advantages:

  • Low learning curve, it acts very similarly to how users are used to operating the mice

• Disadvantages:

  • Difficult to move the cursor across large or multiple displays

Quadrant Jumping 1

• Advantages

  • No. of key presses scales logarithmically with screen area
  • If mastered I think this is probably the fastest method to move the cursor

• Disadvantages

  • Probably the hardest to learn

Quadrant Jumping 2

• Advantages

  • No. of key presses scales logarithmically with screen area
  • Concept of dynamically generated "hints", because each screen chunk is assigned a hint text

• Disadvantages

  • Have to visually parse the hints several times if the hints are dynamic rather than static (e.g. H is always for top-left).

Number one makes more sense here, I think, since speed is what we're after. I agree with blackketter's assessment of hint mode, but still think that jumping to approximately the right area would be faster with a quadrant + movement key combo rather than hints. Hints are kind of overwhelming and for large monitors that'd be quite a few hints. Though maybe it is? 3^26 is 2,541,865,828,329 which is probably plenty, but... idk wouldn't we take a huge performance hit generating these every time? Unless we standardized it and cached them after running it once on a new monitor... idk, something to think about for sure. :/

[blackketter]

I agree with blackketter's assessment of hint mode, but still think that jumping to approximately the right area would be faster with a quadrant + movement key combo rather than hints.

What I was suggesting was that there be a relatively small number of hints in grid mode (maybe 10x10, possibly less, and user tunable). Then you could jump to the correct 1% of the screen with two key hits.

I guess it's a little like quadrants 2, but rather than 2x2 you use the hint-style of jumping using a couple of letter keys to one of 100 spots on the screen. Another way to look at it as two letter hints that jump to row and column.

And that once you do this, you can subdivide that section again to get to a smaller area.

As far as clutter goes, as a comparison, the single Safari window that I'm typing this in now has about 100 hints.

And these two approaches could certainly coexist.

p.s. I've tried to learn quadrant 1 style jumping and I couldn't. I'm already good at typing hints. :)

[dexterleng]

I tried out Voice Control's show grid feature (quadrants 2). Some observations:

• Pretty much every single time the grid has to be drilled into 3-4 times (2 * 4 hint characters to type) until the point I would like to click on appears as the center of a grid cell
• Because I had to expand the grid cells several times, the grid cells became really small, and Voice Control shows a zoomed in view so that the hint text can still be visible:

[dexterleng]

If I have to pick one fallback option, I would pick quadrant 2 because:

1. similar to how the default hint mode works
2. similar to voice control (show grid vs fallback, show numbers vs the current hint mode).
3. low learning curve
4. can show grids across multiple displays

[dexterleng]

Proposal: Workflow for clicking an item (with Fallback)

1. User Activate Hint Mode
2. Hints populate the screen for actionable elements
   • Only home row keys (asdfghjkl) are used
3. User sets relevant action attributes:
   • set count: 0-9
   • toggle modifiers: cmd/shift/option/fn/ctrl
   • set mice click type:
     • c: left click (default)
     • m: move
     • r: right click
   • lock/continuous flag: u
4. if user does not see the hints for the item they would like to click:
   • transition to grid/quadrant-based hint mode with q
     • action attributes are carried over
5. else, type out hint text to perform action
6. if continuous is enabled (default should be disabled), hint mode is reactivated

Hint Keys (home row) do not intersect with Action Keys (non-homerow) because we immediately show hints to users upon activation.

A little Keycastr-esque HUD can be shown in the bottom-right of the active screen for a short period when the user sets action attributes, but fades out after ~1s to avoid blocking hints.

Let me know your thoughts!

[blackketter]

Sounds great! Can't wait to try it!

Will you be able to include the arrow keys to tweak the position of the cursor? Or is that a separate task? (I think you'll need a key to trigger a click in that case. I suggest space.)

I wasn't familiar with keycastr. Nice solution for the HUD!

[dexterleng]

Will you be able to include the arrow keys to tweak the position of the cursor?

What do you like about the cursor approach compared to the grid hints?

[blackketter]

What do you like about the cursor approach compared to the grid hints?

I think both are useful and don't conflict.

The arrow keys are good for incremental mouse movement. Press (and hold) an arrow key and the mouse pointer moves in that direction. That's simple and intuitive.

I think that a grid of hints is a good fallback for movement when an object-based hint isn't available. That fits well with the rest of Vimac and will be quick for folks to learn. Hopefully, over time, Vimac will get better and better at finding targets so that this will be needed less. (I'm still looking forward to hints for the dock, non-active windows and menubar without active windows. :)

[michaelmroczka]

Proposal: Workflow for clicking an item (with Fallback)

1. User Activate Hint Mode

2. Hints populate the screen for actionable elements

   • Only home row keys (asdfghjkl) are used

3. User sets relevant action attributes:

   • set count: 0-9

   • toggle modifiers: cmd/shift/option/fn/ctrl

   • set mice click type:

     • c: left click (default)
     • m: move
     • r: right click

   • lock/continuous flag: u

4. if user does not see the hints for the item they would like to click:

   • transition to grid/quadrant-based hint mode with q

     • action attributes are carried over

5. else, type out hint text to perform action

6. if continuous is enabled (default should be disabled), hint mode is reactivated

Hint Keys (home row) do not intersect with Action Keys (non-homerow) because we immediately show hints to users upon activation.

A little Keycastr-esque HUD can be shown in the bottom-right of the active screen for a short period when the user sets action attributes, but fades out after ~1s to avoid blocking hints.

Let me know your thoughts!

This proposed workflow sounds awesome. I especially like it since it doesn't block other ideas mentioned here. For instance, if Vimac knows you're in "continuous click mode" you could implement the dot operator to repeat-click in the last place you clicked. It'll give you the option to pick from hints still, but if you know that you want to click in the exact same place you could just keep spamming the period button until you've clicked something as many times as you've needed too. And if you then need to move your mouse elsewhere continous mode is still active with hints at-the-ready. Love it.

Also, on a totally unrelated note, I think this product might gain more traction if we renamed this something that wasn't so close to the popular "MacVim". Anytime I'm booting up a new machine and trying to install this I inevitably spend 20 minutes trying to find your repo because google is convinced I want MacVim and not Vimac. Food for thought.