Accessibility: Support text input widgets

[mwcampbell]

A major missing feature in Slint's accessibility support is exposure of text input widgets. I recognize that this is a major design challenge. For AccessKit (i.e. the winit backend), this requires a child AccessKit node for each span of text that has distinct formatting (at minimum, each physical line); see egui for a real-world example of how this is done. Let me know if you need help with this.

[tronical]

Right now we don't support different formatting for spans of text, so this might be easier at the moment.

All text input right now is handled by TextInput items internally, so this might not be too hard to do. Something like this in build_node_without_children (untested):

        if let Some(text_input) = item.downcast::<i_slint_core::items::TextInput>() {
            builder.set_role(Role::InlineTextBox);

            // Get cursor and anchor, text for rendering, etc.
            let visual_representation = text_input.as_pin_ref().visual_representation(None);
            let text = &visual_representation.text;
            builder.set_value(text.clone());
            builder.set_character_lengths(
                text.chars().map(|ch| ch.len_utf8() as u8).collect::<Vec<_>>(),
            );

            todo!()
        }

Or would this have to be a child of a Role::TextField?

[tronical]

We've started work in trying to use cosmic-text, which might also be a good candidate for rich text. When that comes into focus, then perhaps access kit support directly in cosmic-text might be of much value for any cosmic-text user.

[mwcampbell]

Yes, the Role::InlineTextBox node has to be a child of a Role::TextField node. The text field node must be the one that has the focus when the edit widget has the keyboard focus.

[tronical]

@mwcampbell I'm curious, what's a good way to test this so that I know it's done well enough? So far I've used the screen readers in macOS, Windows, and orca on Linux. Do they have a good mode to introspect an inline text box with text selection?

[mwcampbell]

First, AccessKit's AT-SPi implementation doesn't yet include text editing support. We'll get to it soon.

On macOS, Apple has a tool called Accessibility Inspector. I believe it's part of Xcode. And on Windows, there's Accessibility Insights.

I prefer to test with an actual screen reader, but that requires being proficient enough with a screen reader to know what should happen as you interact with the text edit control.

[hunger]

@mwcampbell: I find testing accessibility to be the biggest challenge in supporting it. I did have a blind person introduce me to how he uses a computer with a braille display and a screen reader, but that was hard to follow for me. I was totally amazed that he was able to read anything from that braille line he had! I could hardly fell the knobs at all.

I used Accerciser on Linux when getting the Qt accessibiity started: I tried to make the reports on Slint similar to those of other Linux applications. I think I could not use any of the applications with just the information in the inspection tool, but I am not sure that is because I missed something or because the state of accessibility on Linux in general.

Do you happen to know any good material I could look into to improve my understanding of the topic?

[tronical]

First, AccessKit's AT-SPi implementation doesn't yet include text editing support. We'll get to it soon.

That's good to know.

On macOS, Apple has a tool called Accessibility Inspector. I believe it's part of Xcode. And on Windows, there's Accessibility Insights.

I had tried Xcode's Accessibility Inspector, but I couldn't get it to work yet. Somehow the tree is always empty.

Thanks about the hint about Accessibility Insights, I'll give that one a try - looks very promising.

[mwcampbell]

I finally have an answer to the problem with Accessibility Inspector. It turns out that an application's windows are invisible to Accessibility Inspector unless the app is run from a proper .app bundle. I don't know why, but I confirmed experimentally that that does make all the difference.

[tronical]

Oh that’s good to know. Thanks for finding out!

[mwcampbell]

Just wondering if there's anything I can do to unblock work on this issue. Once accessibility is implemented for text editing, I think slint could easily become the most robust current Rust GUI toolkit with usable accessibility support.

[mwcampbell]

In case this issue was blocked by lack of text support in AccessKit's Unix (AT-SPI) backend, that's now fixed, thanks to @DataTriny.

[RastislavKish]

It's been some time since the last activity on this issue. I'm curious, has there been any progress regarding Slint text input a11y?

AccessKit by now has a well-working cross-platform support for text entries. @DataTriny you've been doing great work on Slint accessibility, are there currently any blockers for the text input?

[DataTriny]

Thanks @RastislavKish for your continued support on my accessibility endeavors.

The last improvement related to text input probably was my pull request to add support for placeholder text.

I don't see major blockers to expose the content of text input widgets to AccessKit. What makes it difficult though is that Slint core only have very limited information on what is inside text inputs, it just has a plain old string as far as I know. More detailed information on characters is stored on each renderer.

AccessKit needs to know the following:

• the Y coordinate of each line,
• the length in bytes of each character,
• the width in pixel of each character,
• the X coordinate of each character,
• the length in characters of each word,
• the line and character index of the start and end of selection.

The simplest approach would probably be to have renderers expose a new method that would compute all of this information. The AccessKit backend would be able to call this to populate its nodes.

I don't think we would need to design something accessible-backend agnostic because Qt doesn't let you create your own accessible text input as far as I know, you just use the built-in one.

@tronical, @ogoffart given the requirements laid above, how would you see the API evolve to accomodate this usecase? Can we avoid rebuilding the whole text input tree everytime its content gets updated? Where would it make sense to put some caching so that renderers and AccessKit share the same representation?

Each span of text (each line of text if we are not talking about rich text) having to be its own AccessKit node, we may have to introduce a new component.

[tronical]

The simplest approach would probably be to have renderers expose a new method that would compute all of this information. The AccessKit backend would be able to call this to populate its nodes.

Right, that sounds doable. We already have similar functionality exposed for the cross-renderer text cursor handling, so adding functionality like this seems very reasonable.

Can we avoid rebuilding the whole text input tree everytime its content gets updated?

That may be difficult. We don't really have a good data structure underneath to support that. I think we'll have to do without that for now.

[tronical]

I'm curious, has there been any progress regarding Slint text input a11y?

Unfortunately no. I don't think that there are any blockers, but it's a bit of work to expose the necessary data (structures) in the renderers to access kit.

[DataTriny]

Two points @tronical:

• For AccessKit, each span of text must be a node with Role::TextRun and these nodes must be direct children of the text input node. Since the renderers operate on the built-in TextInput elements, I think we will have to update the widgets so that all accessible properties are set on the built-in element itself rather than the enclosing one. How would the AccessKit backend find on which elements to ask for textual representation otherwise? It's easy to downcast to TextInput, but it seem brittle to just add the resulting text spans to the tree of the parent of this element. There is nothing to stop someone using raw TextInput in their UI.
• We must assign NodeIds to these text runs. AccessKit NodeIds are currently composed of a component index (32 bits) and the item index (also 32 bits). In the case of TextInput, I assume that every instances will have the same first 32 bits and only the least significant bits will differ. How about making the component index 16 bits, the item index 24 bits so that we still have 24 bits for text span indices? Index 0 would have to point to the TextInput themselves. We could have this pattern only for the TextInput component and keep 48 bit item indices on all other cases if 24 bit item indices feel too small for stuff like ListItems.

[tronical]

Excellent points. Full agreement on both. It's a bit of work, but can be broken down into pieces.

[DataTriny]

I think we will have to update the widgets so that all accessible properties are set on the built-in element itself rather than the enclosing one

This introduces a new issue though as users would not be able to set accessible properties on their widgets, like we set accessible-label on LineEdit widgets in the CRUD example.

The compiler forbids setting accessible properties on widgets with no role, so here is what I currently do to propagate user-defined properties on the LineEdit widget:

diff --git a/internal/compiler/widgets/common/lineedit-base.slint b/internal/compiler/widgets/common/lineedit-base.slint
index 5875ff13b..1f9bf5223 100644
--- a/internal/compiler/widgets/common/lineedit-base.slint
+++ b/internal/compiler/widgets/common/lineedit-base.slint
@@ -22,6 +22,8 @@ export component LineEditBase inherits Rectangle {
     callback key-pressed(event: KeyEvent) -> EventResult;
     callback key-released(event: KeyEvent) -> EventResult;

+    accessible-role: none;
+
     public function set-selection-offsets(start: int, end: int) {
         text-input.set-selection-offsets(start, end);
     }
@@ -63,7 +65,7 @@ export component LineEditBase inherits Rectangle {
         font-family: text-input.font-family;
         color: root.placeholder-color;
         horizontal-alignment: root.horizontal-alignment;
-        // the label is set on the LineEdit itself
+        // accessible-placeholder-text is set on TextInput instead
         accessible-role: none;
     }

@@ -96,8 +98,18 @@ export component LineEditBase inherits Rectangle {
             vertical-alignment: center;
             single-line: true;
             color: root.text-color;
-            // Disable TextInput's built-in accessibility support as the widget takes care of that.
-            accessible-role: none;
+
+            accessible-role: AccessibleRole.text-input;
+            accessible-enabled: root.enabled;
+            accessible-label: root.accessible-label;
+            accessible-description: root.accessible-description;
+            accessible-value <=> text;
+            accessible-placeholder-text: text == "" ? placeholder-text : "";
+            accessible-read-only: root.read-only;
+            accessible-action-set-value(v) => {
+                text = v;
+                edited(v);
+            }

             cursor-position-changed(cursor-position) => {
                 if cursor-position.x + self.computed_x < root.margin {

diff --git a/internal/compiler/widgets/cosmic/lineedit.slint b/internal/compiler/widgets/cosmic/lineedit.slint
index 85ecfa83c..02c97d1d4 100644
--- a/internal/compiler/widgets/cosmic/lineedit.slint
+++ b/internal/compiler/widgets/cosmic/lineedit.slint
@@ -18,12 +18,6 @@ export component LineEdit {
     callback edited <=> base.edited;
     callback key-pressed <=> base.key-pressed;
     callback key-released <=> base.key-released;
-    accessible-role: text-input;
-    accessible-enabled: root.enabled;
-    accessible-value <=> text;
-    accessible-placeholder-text: text == "" ? placeholder-text : "";
-    accessible-read-only: root.read-only;
-    accessible-action-set-value(v) => { text = v; edited(v); }

     public function set-selection-offsets(start: int, end: int) {
         base.set-selection-offsets(start, end);
@@ -54,6 +48,7 @@ export component LineEdit {
     min-width: max(160px, layout.min-width);
     min-height: max(32px, layout.min-height);
     forward-focus: base;
+    accessible-role: none;

     states [
         disabled when !root.enabled : {
@@ -79,6 +74,8 @@ export component LineEdit {
                 text-color: CosmicPalette.foreground;
                 placeholder-color: CosmicPalette.placeholder-foreground;
                 margin: layout.padding-left + layout.padding-right;
+                accessible-label: root.accessible-label;
+                accessible-description: root.accessible-description;
             }
         }

There are a few issues with this approach:

• I am almost certain that the fact the compiler allows setting accessible properties when the role is explicitly set to none is a bug, if yes then we should not rely upon this.
• It require propagating all accessible properties that we expect the users will want to override, here I went with label and description but this might not be enough.
• The resulting AccessKit node for the TextInput will always contain the label and description fields: if they are not set by the user then we will get Some("") even though what we really want is None.

[DataTriny]

I'm trying to go with the approach described above to expose more kinds of widgets by checking the TextInput::single_line and TextInput::input_type properties in the AccessKit backend. This require updating some tests like this one:

diff --git a/tests/cases/widgets/textedit.slint b/tests/cases/widgets/textedit.slint
index 8118b9a81..cd15a8069 100644
--- a/tests/cases/widgets/textedit.slint
+++ b/tests/cases/widgets/textedit.slint
@@ -93,7 +93,7 @@ assert_eq!(instance.get_text(), "Xxx");
 instance.invoke_paste();
 assert_eq!(instance.get_text(), "XxxHello👋");

-let mut edit_search = slint_testing::ElementHandle::find_by_element_id(&instance, "TestCase::edit");
+let mut edit_search = slint_testing::ElementHandle::find_by_element_type_name(&instance, "TextInput");
 let edit = edit_search.next().unwrap();
 assert_eq!(edit.accessible_read_only(), Some(false));
 instance.set_read_only(true);

And I don't know whether this is a bad thing. At some point the TextEdit widget will also expose its scrollbars in the accessibility tree so one could argue that reading top-level accessible properties don't make sense since it is a compound widget. On the other side it seem reasonable to let users customize the label, the description and maybe later other accessible properties that are not mapped to regular properties of widgets.

[tronical]

Hmm, you're raising many good points about the difficult of this:

I think we will have to update the widgets so that all accessible properties are set on the built-in element itself rather than the enclosing one.

Perhaps we need need another delegation property instead, similar to accessible-delegate-focus. Maybe accessible-delegate-node: the-inner-thats-the-textinput and that says that all accessible-* properties are to be taken from the widget but the actual access kit node is to be the-inner-that-the-textinput (and any any accessible-* not set by the widget are used from that one).

[DataTriny]

@tronical This idea also crossed my mind, but there is one point that I cannot solve. Ideally we'd want this new property to be replaced by appropriate property bindings at compile time right? In practice we can expect the users to also use bindings when they set accessible properties on the top-level widget, how would these bindings still resolve at runtime?

Code taken from the CRUD example:

filter-label := Text {
    text: "Filter prefix:";
    vertical-alignment: center;
    horizontal-alignment: right;
}

LineEdit {
    text <=> root.prefix;
    edited => { root.prefixEdited() }
    // How to access filter-label?
    accessible-label: filter-label.accessible-label;
}

Also, we would have to make sure this new property can be used on multiple levels: in the case of TextEdit (except for the cupertino style), we would have to apply it on TextEdit (pointing it towards TextEditBase and then on TextEditBase (making it point to TextInput).

There is definitely an interesting challenge to solve here.

[DataTriny]

Nevermind, figured this out by myself: for each element referenced by this new property, the compiler will have to set all accessible properties on the referenced element that are set on the root property, making them aliases to these accessible properties defined on the root element. We will have to allow setting accessible properties on elements if they only set this property but otherwise don't have an accessible role. Similarly, the valid elements that should be allowed for this property would be those that either have an accessible role or that also define this property.

This should probably be a new compiler pass, and I think the focus_handling pass can be used as inspiration.

Most important question: which name to go with for this new property?

• accessible-delegate
• forward-accessibility
• Something else?

[DataTriny]

So I gave this a try but quickly ran into issues. What makes this more complicated is that Element::is_binding_set only considers bindings that are defined inside the element itself. To know which accessible properties need to be forwarded, I guess we would have to visit the entire tree of the parent element (excluding self).

component Inner {
    accessible-role: button;
}

component Outer {
    forward-accessibility: inner;

    inner := Inner { }
}

export component Test inherits Window {
    in property enabled <=> outer.accessible-enabled;

    outer := Outer {
        accessible-label: "A button";
        accessible-checkable: true;
    }

    public function check() {
        outer.accessible-checked = true;
    }
}

In the example above, ideally we'd want to apply accessible-enabled, accessible-label, accessible-checkable and accessible-checked to the inner element. But when we detect that outer has forward-accessibility, we can't easilly find these bindings from within outer. We'd have to go to its parent and look on the properties, inside functions and on Outer's instanciatiation to find them.

@tronical I'll need your knowledge of the compiler here. Is this what you had in mind? Does it seem viable to visit the parent to check which bindings to consider, or is it going to be too heavy?