Open Utau Accessibility Improvements

[dangero2000]

trafficstars

Acknowledgement

• [X] I have read Getting-Started and FAQ

🐛 Describe the bug

I am a screen reader user due to having almost 0 eyesight. Unfortunately, Open Utau is not currently accessible with screen reading software, but I'm hoping we can work together to change that. I have read the Getting Started and FAQ pages, and from those I have come up with a basic idea of where we can start. There are some keyboard shortcuts for actions already, but not every action is accessible with the keyboard and requires the use of a mouse. Actually, most of the actions pointed out in the FAQ are exclusive to the mouse. I think, before adding speech with a screen reader, this would be an amazing place to start. If Open Utau can be used exclusively with the keyboard, you’re already half way to making the program fully accessible. If you have any questions for me, please do let me know, and I’ll try to answer them to the best of my abilities. Let’s make the music world more accessible together!

Explains how to reproduce the bug

N/A

OS & Version

N/A

Logs

N/A

[lselden]

I've been looking at this, as I myself would like to use the keyboard for navigating OpenUTAU more. I'm not ready to open a pull request, but my initial attempts have been promising - I've gotten basic piano roll navigation and open/closing the edit lyrics window working. While I'm poking at it I want to put down some development notes, hoping someone more familiar with the OpenUtau code would want to comment on best approaches.

Dev Notes

• Architecturally I'm thinking of creating a global key manager, which looks at the currently active view is and send the appropriate action as necessary, similar to how Visual Studio Code handles key bindings. Would it make sense to put that sort of shared functionality in a new folder/namespace of OpenUtau?
• One way to support keyboard navigation as well as mouse nav menu items need to use Commands rather than the Click event handler. This assumes that the Command/Method is on the ViewModel instead of code-behind (unless someone knows how to target the code-behind using Binding). This also allows using "Accelerators" - just put an underscore (_) before the letter used for triggering the menu item. For example, "Open" becomes "_Open". Then a user can press Alt+F to open "_File", then O for the Open option.
• The HotKeyManager built into Avalonia only supports 1 KeyGesture per UI element, so you can't bind multiple KeyGestures per command. That's a downer, because even just supporting Redo (Ctrl+Y or Ctrl+Shift+Z) can't happen. Also it would mean adding a big menu with each of the available commands (move selection left, etc) - that's okay, but seems annoying.
• Alternatively, we can keep useing the existing OnKeyDown handlers, along with InputGesture on MenuItem entries.
• The latest Stable version of Avalonia (10.18) includes a KeyGesture class and some helpers for detecting the common keystrokes (copy/paste/move) - I propose using a common event map using this similar to this code.

Benefits

• Using a shared global set of Commands instead of Events would make API automation much easier - You could add in OSC listeners for example, or a HTTP API Server that dispatches the commands for transport, add notes, etc. (I get that that should be in the Core code not UI, but common features like "move play position 1 bar forwards" could be shared).
• Editing experience with keyboard is fun! I've gotten selecting next/prev note and adding a new note to the end of Part done, and binding / to open up the "Edit Lyrics" menu, and it's so much faster for me to enter in notes.
• Accessibility, of course.

How you can help

• We need a UX design of how to actually navigate and edit content. I've proposed a start below, but I'm coming from a video editing / programming background, so others may have different preferences. Any help in coming up with key bindings and what gestures/actions are needed are appreciated.

• I have no experience with internationalization, so I don't know if the keybindings need to change for different languages.

• I'm not going to get to reading the preferred keybindings from a config file or something, but it would be nice to be edit-able

• This site may be helpful for seeing what sort of conventions are out there.

• I have no idea how expressions / velocity curves could be edited via keyboard...I guess you could add selection functionality per point, and navigate like you would notes?

Questions

• How aggressive can I edit the OpenUtau MVVM code without stepping on other people's toes?
• Is there a good reason for "Edit Lyrics" text box to capture Tab/Return? I can't imagine so, but it breaks keyboard navigation.
• Should there be a separate Tool in the piano roll for keyboard-based editing, or should it be a toggle option?
• Can adding new notes default to the tone C5 (no notes), or copying the previous/selected note's pitch? Or do we need some sort of computer keyboard -> piano detection?
• Are the numpad keys a preferred way to move around, or stick to directional arrows + regular keyboard?
• When navigating forward/backward in the Piano Roll / Part view should the primary navigation be by playhead (i.e. arrow keys move playhead forward/backward) or by selection (i.e. arrow keys move selection to next/prev note/part)

Suggested key bindings

Action	Keys
Undo/Redo/Cut/Copy/Paste	Standard Keys
Select Next/Prev Note	← / →
Transpose Selection 1 step	↑ / ↓
Transpose Selection 1 octave	Ctrl+↑ / Ctrl+↓
Extend Selection	Shift + → / Shift + ←
Extend Selection To Beginning / End of Part	Shift + Home / Shift + End
Select All	Ctrl + A
Select None	Ctrl + D
Move Selected Note 1 Snap Point	[ / ]
Move Selected Note 1 Bar	Ctrl+[ / Ctrl+]
Change Duration of Selected Note 1 Snap Point	Alt+[ / Alt+]
Move Playhead 1 bar	PageUp / PageDown
Move Playhead 1 Snap Point	Alt+PageUp / Alt+PageDown
Move Playhead 1 window length (based on zoom)	Ctrl+PageUp / Ctrl+PageDown
Move Playhead to beginning/end of Part	Home / End
Zoom In/Out Horizontally	+ / - (actually + is the the equals key)
Zoom In/Out Vertically	Alt + + / Alt + -
Scroll window 1 window length	Alt+Home / Alt+End (? I don't know about this one)
Center scroll on current selection	Ctrl + \
Open "Edit Lyrics"	/
Apply "Edit Lyrics" changes and close	Enter Breaking Change
Cancel "Edit Lyrics" changes and close	Esc
Play/Pause	Space / Space Breaking Change - currently goes back to beginning of part
Play/Stop	Space / Ctrl+Space (Existing functionality)
Switch to part in next/prev Track	Ctrl + ; / Ctrl + ' (??? I'm not sure of this one, but I'd love to be able to edit other voices singing in harmony. We'll need to account for parts starting at different times)
Focus on Main Window	F5 (? not sure)
Focus on Piano Roll	F6 (? not sure)
Other Functionality	??? What's missing?

[stakira]

It's very appreciated you put your thoughts into improving OpenUtau in this area. I'll reply in line.

But before going into details. Accessibility involving screen reading is an area I feel daunted about, and honestly don't have time to get into. But as the OP pointed out, the keyboard shortcuts are a good starting point. I think this area is better understood generally and within reach. I'm happily open to discussion and PR.

• Architecturally I'm thinking of creating a global key manager, which looks at the currently active view is and send the appropriate action as necessary, similar to how Visual Studio Code handles key bindings. Would it make sense to put that sort of shared functionality in a new folder/namespace of OpenUtau?

Practically speaking, each window has its own shortcuts and should be handled within the window. I don't get the point of making them global. The current pattern with OnKeyDown() seems simple and sufficient.

• One way to support keyboard navigation as well as mouse nav menu items need to use Commands rather than the Click event handler. This assumes that the Command/Method is on the ViewModel instead of code-behind (unless someone knows how to target the code-behind using Binding). This also allows using "Accelerators" - just put an underscore (_) before the letter used for triggering the menu item. For example, "Open" becomes "_Open". Then a user can press Alt+F to open "_File", then O for the Open option.

Not everything is doable via command, e.g., opening another window. A command creating a window is possible but painful and against MVVM principle. I wouldn't base the design on using command entirely.

Accelerator is good but hard to manage for all the translations. I used them before but removed to keep things simple. Sorry but I don't think that's very manageable going forward.

I made the majority of non-menu widgets unfocusable, which is unfortunate for screen reading. The reason is such focusing often gets in the way of Space shortcut.

• The HotKeyManager built into Avalonia only supports 1 KeyGesture per UI element, so you can't bind multiple KeyGestures per command. That's a downer, because even just supporting Redo (Ctrl+Y or Ctrl+Shift+Z) can't happen. Also it would mean adding a big menu with each of the available commands (move selection left, etc) - that's okay, but seems annoying.
• Alternatively, we can keep useing the existing OnKeyDown handlers, along with InputGesture on MenuItem entries.
• The latest Stable version of Avalonia (10.18) includes a KeyGesture class and some helpers for detecting the common keystrokes (copy/paste/move) - I propose using a common event map using this similar to this code.

InputGesture is only for display, which we should actually use. HotKey simply doesn't work when I tried. So I guess eventaully OnKeyDown() is straightfoward and works. In general, I don't think we need to follow the frameworks or design pattern too strictly. Sometimes they could be over-engineered for our use case.

In suammy: what I think actionable is:

• Design and implement more shortcuts using OnKeyDown().
• Find a way to allow focusing on widgets yet still supporting Space.

Questions

• How aggressive can I edit the OpenUtau MVVM code without stepping on other people's toes?

Please do small PRs. If you try to limit changes to local, it's probably fine.

• Is there a good reason for "Edit Lyrics" text box to capture Tab/Return? I can't imagine so, but it breaks keyboard navigation.

Tab to next note and enter to commit are shortcuts used by lyrics box. Without checking it's possibly a bug that it's capturing keys when it's invisible.

• Should there be a separate Tool in the piano roll for keyboard-based editing, or should it be a toggle option?

Ideally it should not conflict with other shortcuts. Then we just leave it always on.

• Can adding new notes default to the tone C5 (no notes), or copying the previous/selected note's pitch? Or do we need some sort of computer keyboard -> piano detection?

Not really a problem if user can adjust using up / down to adjust, and hear the tone upon creation and move.

• Are the numpad keys a preferred way to move around, or stick to directional arrows + regular keyboard?

Numpad is getting rare these days. Maybe avoid them.

• When navigating forward/backward in the Piano Roll / Part view should the primary navigation be by playhead (i.e. arrow keys move playhead forward/backward) or by selection (i.e. arrow keys move selection to next/prev note/part)

Maybe depends on if any note is already selected? These have clear feedback, so user should have no problem to learn to deselect first.

Suggested key bindings

I'd like to suggest design them in groups:

• Note selection. I think your choices are intuitive, with little-to-none learning curve. Adding a few extra here.
  • Left / Right
    • When nothing is selected, starts selection with the note in center of view
    • Or move single note selection
    • Or move and select "head", discarding range selection
  • Shift + Arrow / Home / End / Page Up/ Page Down to range select. This works like windows explorer or text editors: selection always changes at the "head", i.e., if expanding to the left with ←, → should shrink the selection, unless the selection is shrinked to a single note and starts expanding to the right, then the "head" is now the right-most note, → becomes expanding and ← becomes shrinking.
  • Esc to cancel selection if only one note is selected. Or shrink selection to the "head". Note that as a result double Esc consistently cancels any selection. Easy to learn.
  • Pretty much like Windows explorer or text editors, though "head" is not visibly cued.
  • View should move when "head" goes out of screen.
• Moving notes
  • (Ctrl) + Up / Down to transpose like today.
  • Ctrl + Left / Right to move.
• Changes note / part length with - / +(=).
• Playhead
  • [ / ] for small moves
    • Page Up / Down / Home / End are not on compact keyboards so I don't want to rely on them for essentials.
  • Ctrl + [ / ] to move to start / end of current selection
  • Shift + [ / ] to move to start / end of current view
  • Page Up / Down / Home / End for big moves (Fn + Up / Down / Left / Right on mac)
  • View should move when playhead goes out of screen
• Moving view only
  • WASD for moving view. F to put selection in center. Please see oto view shortcuts. QE zooms instead.
  • I think we don't need to worry about zooming vertically since they are not that useful.
• Lyrics
  • With a note selected, Enter opens LyricBox
  • Esc to cancel. Enter to commit and close: already like this today. If not then it's a bug. Are you on mac?
  • Tab commit and to next note. Shift + Tab commit and to prev note: already like this today. If not then it's a bug.
• Moving between windows: "alt + tab" is enough for now.

[lselden]

Thank you for your comments. I like your suggestions. I'll work on adding in as small PRs.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further activity occurs. Thank you.