NVDA/JAWS/Voiceover reads content multiple times on focus

[Leon-Lj]

Description

When you focus on a math-field that has content in it, NVDA will read the content twice (one for aria-live, and once for the label), JAWS will read it 4 times (aria-live, mathml, label, aria-live), Voiceover will read it once if you click on the math-field (aria-live I think) but twice if you tab into the math-field (aria-live, label).

NVDA can't read mathml (at least not out of the box), but Voiceover can so I'm not sure why it doesn't try to read the mathml as well. Maybe Voiceover is smart enough to not repeat itself.

JAWS reading aria-live multiple times seems like a bug in JAWS.

aria-labelledby is supposed to take precedence over aria-label (according to mdm. So in theory we should be able to set both and have aria-labelledby point to the mathml.

    markup.push(
      `<span contenteditable=true aria-labelledby=mathmlId aria-multiline=false class=ML__keyboard-sink autocapitalize=off autocomplete=off autocorrect=off spellcheck=false inputmode=none tabindex=${
        element.tabIndex ?? 0
      }></span>`
    );
...
this.accessibleMathML.setAttribute('id', 'mathmlId');

This helps a bit, JAWS won't read the label, but now it'll read the mathml twice. While Voiceover won't read the mathml (and only reads the label if you use tab to focus).

If I add this.accessibleMathML.setAttribute('aria-hidden', 'true') then JAWS will only read the mathml once, and skip the label. But that seems wrong, I shouldn't hide a aria-labelledby element, should I?

And of course, NVDA will also stop reading the label, and try and fail to read the mathml. So it does not work well there.

I haven't been able to find a way that works for all three. I can't get Voiceover to work without aria-live, but with it NVDA/JAWS will read the content multiple times.

Best solution for JAWS/NVDA seems to be to:

1. Disable aria-live for focus event, but still set the label (lines 85 and 86 in a11y.ts)

} else if (action === 'line') {
    const label = speakableText(mathfield.options, '', mathfield.model.root);
    mathfield.keyboardDelegate.setAriaLabel(label);
}
...
if (liveText) {
    // Aria-live regions are only spoken when it changes; force a change by
    // alternately using nonbreaking space or narrow nonbreaking space
    const ariaLiveChangeHack = mathfield.ariaLiveText.textContent!.includes(
      '\u00a0'
    )
      ? ' \u202F '
      : ' \u00A0 ';
    mathfield.ariaLiveText.textContent = liveText + ariaLiveChangeHack;
}

2. Remove the mathml (technically for JAWS it's better to have it there properly labeled, but then NVDA won't work).

Best solution for Voiceover, maybe the way it's currently is? Although it is odd that it behaves differently depending on if focused by tab or mouse click.

All testing here was done with mathlive 0.89.4 using Mathlive Smoke Test.

Environment

MathLive version 0.89.4

Operating System Windows 10, 21H2 (OS Build 19044.2486) macOs Ventura 13.2.1

Browser Chrome 109.0.5414.120

Screen readers NVDA - 2022.4 (2022.4.0.27401) running on Chrome JAWS - 2023.2302.15 running on Chrome VoiceOver

[NSoiffer]

Removing the MathML is the wrong fix -- you lose the braille. If AT follows the ARIA spec, then using aria-label (and aria-labeledby) should override the braille that would otherwise be generated, so you don't get proper braille for the math. I suspect the proper fix is somehow to just the MathML. Only use the aria-live region for announcing edits and moving in the mathfield. But I haven't done the testing that @Leon-Lj has done.

[arnog]

Not clear to me that there is a fix that would cover all use cases. Seems like it's a choice between supporting braille output or NVDA. I'm open to suggestions, though.

[NSoiffer]

I'm at a conference today, but I'll look into NVDA tonight or on Wednesday. It should be doable as NVDA math works in browsers using the same hack that (I think) mathlive is doing because it is modeled after the way Mathjax does accessibility with MathML.

FYI: although NVDA doesn't come out of the box with math accessibility, anyone who uses math on any regular basis either downloads MathPlayer or adds the MathCAT or Access8Math addons.

[Leon-Lj]

I had not considered braille.

Ideally NVDA should be able to detect that the aria-labelledby is something it does not understand, and fall back to aria-label. So that we could have something like <span aria-label="2 TIMES x" aria-labelledby="mathmlId">. NVDA has been responsive to bug reports in the past, maybe they could be of help?

Isn't the way Mathjax handles accessibility through the Speech Rule Engine?

[NSoiffer]

Isn't the way Mathjax handles accessibility through the Speech Rule Engine?

It depends.

If you use the right click menu, you can activate accessibility, but there are a number of limitations caused by HTML standards of today. One is that there isn't a way to get the braille for the math to the screen reader -- just the speech, and even then, there are problems with the "a" sound and no pausing. In both JAWS and NVDA, there is a braille viewer so you can see what is being sent to a braille display. With accessibility on, you will see it is just the text for the speech. The speech limitations and the lack of math braille is because MathJax/SRE is using aria-label.

If you deactivate MathJax accessibility (off is the default), then the screen reader reads the hidden MathML and generates Nemeth braille. MathCAT offers the option of UEB technical also (required in some states and the standard outside the US -- it's controversial...).

[NSoiffer]

With NVDA, it seems that removing the two aria-labels gets a single reading. I'm not 100% sure because I was using https://cortexjs.io/mathlive/ and the firefox debugger and the aria-label that includes "after: ..." kept coming back after I deleted it. Because it had the word "after" in it, it is clear where that second speech is coming from.

I don't have JAWS on my laptop (flying today), so I can't test with it. But I'm pretty sure the multiple readings are because of the aria-labels and the MathML. I think the labels shouldn't be there. Is there a reason they were added?

[Leon-Lj]

I installed MathCat for NVDA and confirmed that it's properly reading some mathml on a testpage (just to be sure I got it working).

Using the current head of the master branch (commit ee2d82b5aa1d17b2c460e040d549094307e2a557) I modified it to stop setting the label and to bring back the mathml.

In a11y.ts

  } else if (action === 'line') {
    // Announce the current line -- currently that's everything
    // const label = speakableText(mathfield.options, '', mathfield.model.root);
    // mathfield.keyboardDelegate.setAriaLabel(label);
  } else {

In mathfield-private.ts

  readonly accessibleMathML: HTMLElement;
...
    const accessibleNodeID =
      Date.now().toString(36).slice(-2) +
      Math.floor(Math.random() * 0x186a0).toString(36);
    // Add "aria-labelledby="${accessibleNodeID}"" to the keyboard sink

    // 1/ The keyboard event capture element.
    markup.push(
      `<span contenteditable=true aria-multiline=false class=ML__keyboard-sink autocapitalize=off autocomplete=off autocorrect=off spellcheck=false inputmode=none tabindex=${
        element.tabIndex ?? 0
      }></span>`
    );
...
    markup.push(
      `<span class=accessibleMathML id="${accessibleNodeID}"></span>`
    );
...
    this.accessibleMathML = this.element.querySelector('.accessibleMathML')!;

In renderer.ts

  mathfield.accessibleMathML.innerHTML = mathfield.options.createHTML(
    '<math xmlns="http://www.w3.org/1998/Math/MathML">' +
      toMathML(model.root, mathfield.options) +
      '</math>'
  );

Also edited index.html for smoke test to show a simple expression

<math-field id="mf" virtual-keyboard-mode="manual"
        >m=\frac{n}{x}</math-field
      >

This includes the mathml, but does not aria-labelledby it, so it's similar to how it was before.

NVDA with MathCat: Does not read it at all, simply says "section editable blank" Voiceover: Does not read it at all, simply says "group, editable" JAWS: Reads it once "cursive small m equals cursive small n cursive small x" (nothing is shown in the speech viewer interestingly enough)

I then set aria-labelledby=${accessibleNodeID} on the content-editable span.

NVDA with MathCat: The speech viewer claims it's saying "m = n x section editable blank", but I can only hear "equals section editiable blank" Voiceover: Viewer shows "m = n x, group, editable" but I can only hear "equals, group, editable". JAWS: Reads it twice, "m = n x" in viewer reads it as "cursive small m equals cursive small n cursive small x"

So I created a simple test html file using the mathml from the smoke test:

<html lang="en-US">
<body>
  <math id="eq1" xmlns="http://www.w3.org/1998/Math/MathML" display="block">
    <mrow><mi>m</mi><mo>=</mo><mfrac><mi>n</mi><mi>x</mi></mfrac></mrow>
  </math>
  <input type="textbox" aria-labelledby="eq1"></input>
</body>
</html>

NVDA with MathCat: Reads the mathml just fine "m equals n over x", but when I focus in the inputbox "m = n x edit blank". Voicover: Can't get it to read the mathml expression, can just step through the components and it reads the letter but not the fraction. Reads "m = n x edit text" when focus. JAWS: Same as NVDA

It looks to me like neither NVDA nor JAWS can handle aria-labelledby to a mathml element. What I don't understand is how JAWS can read the mathml expression in the test, but not in the first example (whe labelledby isn't used)? Should those not be the same?

Tested on Windows 10, 21H2 (OS Build 19044.2486), Chrome 110.0.5481.178, NVDA 2022.4 (2022.4.0.27401), JAWS 2023.2302.15. macOS Venture 13.2.1.

[NSoiffer]

ARIA has an algorithm for how aria-label (and hence, aria-labelledby) determine the the text for the label. That algorithm does not include knowledge of MathML and likely just gathers up all the children. Hence, you don't hear "fraction". AT probably does something special when aria-label is on MathML. That would account for why the two techniques get different results.

If one hasn't focused on the mathfield, the hidden MathML will get spoken at some point when navigating past the mathfield. Putting an aria-label on the mathfield means there are two places that want to speak it. Did I get that right?

I think what needs to happen for the mathfield is that when focus is there, it needs to use the aria-live region to self-voice the math, just as it does when one edits it. Maybe there is some trick where the focus moves to the MathML so screen readers read the math and then back to mathfield for editing, but that's a bit scary/ugly to me.

Adding `aria-label="math input area" or some such on the mathfield is something to consider, although that likey results in hearing "math input area" followed by the reading of the MathML. So maybe just setting it as an empty string or some similar short string "editable" to get a consistent reading is the way to go.

[Leon-Lj]

The main problem is that when the mathml is included as described above (without using aria-labelledby), which is the same as it was in 0.89.4 then JAWS will read the mathml on focus, but it will read it wrong, while the text viewer shows nothing. For example it reads m=\frac{n}{x} as "cursive small m equals cursive small n cursive small x", aka it does not read it as fraction (but if I place the mthml in the html, outside of mathlive, it reads it just fine). So maybe this is a bug in JAWS?

If I test with NVDA and MathCat it doesn't read it (I can't read braille, but it doesn't look like it's there either), while just like JAWS if the mathml is placed in the html outside of mathlive, it reads it just fine. How do you access the mathml from mathlive to get it to output speech/braille when you test @NSoiffer?

I think the goal should be to as closely as possible match the behavior of a textbox/textarea. NVDA will read and output braille of the content when you focus a textbox/textarea. JAWS will read and output braille of the content for textbox (<input type=text>), but for a text area it will simply say "has content" (but you can put it in "Read all" mode and then it'll read the content on focus).

So, lets assume the desired behavior is to have it read and output braille of the content on focus. There seems to be 3 options:

1. Use mathml, I haven't been able to get it to work. JAWS reads it wrong, NVDA (with MathCat) won't read it at all, unless a aria-labelled is provided that points to the mathml. In which case it reads it wrong. Since reading it wrong I assume that the braille is wrong, but I can't be sure about that.
2. Use aria-live, NVDA will read it fine, but the braille will be of the "wordy kind" aka just the same as the spoken text (once again, I assume). JAWS will for whatever reason read it twice, braille is same as for NVDA.
3. Use aria-lable, NVDA and JAWS reads it fine. Braille is of "wordy kind".

Point 1 can be combined with 2 or 3. But that results in JAWS reading the mathml (reading it wrong) as well as the other method. While NVDA does not read it, but does not seem to be able to take advantage of it either (aka the braille output isn't of the good mathematical kind).

If you want to test with the same code as I, you can find all the different combinations here (based off 37be2c2d6eccdc097bdb7c2312904b04f9383d16): https://github.com/kikora/mathlive/tree/mathml https://github.com/kikora/mathlive/tree/aria-live https://github.com/kikora/mathlive/tree/aria-label https://github.com/kikora/mathlive/tree/aria-live-mathml https://github.com/kikora/mathlive/tree/aria-label-mathml

I have modified Smoke test to include the example expression I've used here as well as showing it (or as close to it) in a textbox, a textarea and mathml.

[Leon-Lj]

Maybe the reason why JAWS reads the aria-live twice is because it's a child node of the <math-field>. If it was moved outside of it, maybe that option could be made to work the best (I think Voiceover works with aria-live)? The same is probably true for the mathml, but if it was moved outside of the <math-field> and we can't use aria-labelledby because it's broken then how would we be able to properly reference it?