termux-app Let screen readers catch cursor position

TErmux is almost usable with a screen reader like brltt.

One major barrier to its use, though, is that brltty does not know where the cursor is. More specifically, no matter her the crr iviually, brlttyshows it at the botom-right of the reen.

Could this please be fixed?

If made usable with brltty, termux could become a fundamental app for visually impaired persons, especially since braille notetakesrs like HIMS Braille Sense 6 are running Android.

Jan 24 '22 22:01 shindere

Hey. The issue is that the TerminalView is not really a text field (EditText) used by android accessibility service and so there is no concept of "cursor position" for it. Consider it more like an image with pixels. Termux app does publish the text shown in the terminal as contentDescription added in #344.

My guess was that cursor being reported at bottom right was because of terminal not being a text field, which I confirmed by installing the brltty android app from https://play.google.com/store/apps/details?id=org.a11y.brltty.android as documented in https://brltty.app/archive/Android/brltty-on-android.html. I enabled logcat logging in accessibility settings and enabled Cursor Tracking log category. Brltty logged the following entries when I opened the terminal with extra keys mode, then switched to text input view mode, then typed 3 characters and then switched back to extra keys mode. The text input view is an EditText, so it reports the cursor position correctly. The cursor position for terminal is always 0,1, but changes to 0,40 in input text view mode and updates when text is changed. There are some extra entries generated during swipe action when switching to text input mode.

The log entries are from brltty/Programs/update.c

W/brltty: cursor tracking: scr=237 csr=[0,1]->[0,47] win=[0,1]->[0,47]
W/brltty: cursor tracking: scr=237 csr=[0,47]->[0,42] win=[0,43]->[0,43]
W/brltty: cursor tracking: scr=237 csr=[0,42]->[0,40] win=[0,41]->[0,41]
W/brltty: cursor tracking: scr=237 csr=[0,47]->[0,40] win=[0,41]->[0,40]
W/brltty: cursor tracking: scr=237 csr=[0,40]->[1,40] win=[0,40]->[1,40]
W/brltty: cursor tracking: scr=237 csr=[1,40]->[2,40] win=[1,40]->[2,40]
W/brltty: cursor tracking: scr=237 csr=[2,40]->[3,40] win=[2,40]->[3,40]
W/brltty: cursor tracking: scr=237 csr=[3,40]->[0,1] win=[3,40]->[0,1]

As for whether this can be fixed or not, I am currently not sure since I don't really have experience with accessibility services. However, there seems to be support in android for custom views as documented at https://developer.android.com/guide/topics/ui/accessibility/custom-views. Maybe custom events can be published with updated AccessibilityNodeInfo.EXTRA_DATA_TEXT_CHARACTER_LOCATION_KEY in View.onInitializeAccessibilityNodeInfo(). Chromium seems to be doing something similar in OWebContentsAccessibility. But not sure if its the right thing for termux terminal.

I will try looking into this. I don't have a braille screen reader device, so that may be an issue too and may need someone who does for testing.

Feb 05 '22 20:02 agnostic-apollo

Many thanks for your thorough investigation!

I do have a braille display and will be happy to test and assist in any possible way!

@sthibaul may have an opinion about the way to go, although I am not sure how familiar he is about accessibility on Android.

Feb 05 '22 20:02 shindere

You are welcome. Cool, will let you know in case I need testing.

Feb 05 '22 20:02 agnostic-apollo

Thanks. Really. For blind people, fixing this issue would be a huge enhancement.

Feb 05 '22 20:02 shindere

Yeah, I understand and hope this gets fixed somehow. I'll try prioritizing this.

Feb 05 '22 21:02 agnostic-apollo

agnostic-apollo (2022/02/05 13:03 -0800):

Yeah, I understand and hope this gets fixed somehow. I'll try prioritizing this.

It's already wonderful you got interested in this and went that far. YOur comment was awesome with all the details and links you provided.

Feb 05 '22 21:02 shindere

I don't know anything about the accessibility stack on Android.

Feb 05 '22 23:02 sthibaul

note there is an alternative accessibility method which is using an actual linux specific console screenreader. had some success with fenrir but it required proot setup and some way for it to contact the external pulseaudio, I don't even remember what I did.

Jun 15 '22 17:06 webczat

The issue reported here is independent of the screen reader at use, I think.

For a screen reader to work, it must have a way to "see" what is displayed on the screen.

When I last tested, it was almost the case with Termux. The only problem I could see during my rough tests was that the reported position for the cursor was wrong: it was constantly reported to be at the bottom-right of the screen, even in Emacs, for instance.

Meanwhile, I got my Android braille display so I am available to perform tests.

Jun 15 '22 18:06 shindere

I was actually looking into this yesterday before you guys messaged.

I have managed to find a way to send terminal cursor position via AccessibilityNodeInfo.extras. There doesn't seem to be any other valid fields that could be used. Moreover, more custom data can be sent as well or specially returned if requested by accessibility apps.

Check #2840. You can download termux apk build from https://github.com/termux/termux-app/actions/runs/2509869570. Note that if you installed termux from F-Droid, then installation github build will fail since it will have different signing keys, so you would need to uninstall termux and all plugin apps first. Check https://github.com/termux/termux-app#Installation.

You would also need to apply the brltty app patch to get the values. Currently, its just logging the values and not setting screen reader cursor. Log Accessibility Events will have to be enabled in advanced brltty settings. Let me know if I should open a pull request for this.

The issue is that I don't have experience with screen readers or terminal support for screen readers and don't know how this should ideally be implemented and how useful cursor position is. The terminal is two dimensional, but screen readers seem to be one dimensional, basically a single row with limited text characters. So there is some mapping going on as well. Moreover, in non text edit mode (outside nano, etc), the user may want to read text at the top of the terminal and cursor would be at the bottom waiting for new commands, so content description would have to start at the top and not at the cursor at command line. In text edit mode, the cursor would be aligned with text to be read, so likely more ideal. But even in text edit mode, when the cursor is at extreme right of terminal, there may still be text available to the right on the line of the file being read, in that case cursor position would still be same, even after horizontal scrolling.

The brltty app is also not considering if text field is multiline and cursor tracking trky is at wrong offset and even returning to text field after app switch seems to be broken.

Are there some standards available that should be supported by terminals for screen readers? There is some info on cursors at http://brltty.app/doc/Manual-BRLTTY/English/BRLTTY-5.html#ss5.1

I need more info on what's expected by a screen reader or a blind person so that an ideal system can be designed since obviously there are issues.

Console screen readers are available too and it might be possible to add native packages for them in termux. There was also work being done audio desktop like emacspeak at https://github.com/termux/termux-packages/issues/6504

Android accessibility also has support for virtual views. So, for the terminal, I should be able to implement something like 3 modes. Possibly extras keys can be used to switch between the modes, but users would have to configure the extra keys, so not sure how convenient that would be at app install.

Window mode in which entire terminal text is set as content description and terminal is considered a single view.
Line mode in which each row is a separate view, allowing users to move between rows and having their text read.
Character mode in which each row and column is a separate view.

Currently, termux is using window mode and setting content description to the entire terminal text that is visible so TalkBalk starts reading text from the top everytime, which would obviously be annoying and time wasting. So line and character mode may be useful as well. It would also have to be decided if view mode should apply to all opened terminal sessions or only current one. I also don't know how screen readers handle incoming text, I am assuming its buffered and can be scrolled through and read separate from android.

Android has a sample for virtual views as well.

https://github.com/Miserlou/Android-SDK-Samples/blob/master/ApiDemos/src/com/example/android/apis/accessibility/AccessibilityNodeProviderActivity.java

Chromium also has extended accessibility support.

https://github.com/chromium/chromium/tree/main/content/public/android/java/src/org/chromium/content/browser/accessibility

Improving accessibility support via Android APIs would require collaboration will brltty devs, possibly @DaveMielke who seem to be handling android side. Any input on above would be appreciated.

Jun 16 '22 15:06 agnostic-apollo

unsure if screenreaders which use speech can easily be made to work well with any kind of terminal on android. I hope it's possible but might not be, as their usage is kinda different compared to desktop screenreaders. Might be far better in case of braille, but I don't have any braille display. Console screenreaders can work in theory, in practice I managed to get a setup using proot archlinux and pulseaudio, it worked but was noticeably slower to be honest. Console screenreaders which are not linux specific often grab a pseudoterminal and don't care about anything else, just intercepting characters coming into or out of a pseudoterminal, so no support from termux is needed in that case except things these screenreaders want to use which normally don't work or things like sound/speech synthesis.

Jun 16 '22 15:06 webczat

Well, proot is going to be slow, hence native support if possible would be better.

Jun 16 '22 16:06 agnostic-apollo

well if any console screenreader would work without proot that would be something too. the one I have in mind probably doesn't.

Jun 16 '22 16:06 webczat

Hi:

Since you're including me in this discussion, I'm assuming that you'd like me to contribute to it. Could you please explain to me what your goal is? That'd help me know which kinds of answers you'd like.

You mention brltty. It's already a fully functional screen reader so what are you trying to offer on top of it?

You mentioned that brltty doesn't work with multi-line text views. That isn't correct. It works fine with them.

Jun 16 '22 19:06 DaveMielke

Dave,

I am the person who initially opened the issue. Are you able to read its content?

In a few words: termux is an Android app which lets you use command-line tools on an Android device. You can install packages, edit files, etc.

When you isntall both termux and brltty on an Android device, brltty already has access to the screen content exposed by Termux. The last time I tested, the only thing brltty couldn't get was the position of the cursor.

How to fix this is what is being discussed here, but it may be helpful if you were able to read the issue from the beginning.

If you are not able to do so in a comfortable way, just let me know and I will send you a text export of everything that has been posted to it.

Jun 16 '22 20:06 shindere

[quoted lines by Sébastien Hinderer on 2022/06/16 at 13:13 -0700]

In a few words: termux is an Android app which lets you use command-line tools on an Android device. You can install packages, edit files, etc.

So all I need to do is to install the termux app?

When you isntall both termux and brltty on an Android device, brltty already has access to the screen content exposed by Termux. The last time I tested, the only thing brltty couldn't get was the position of the cursor.

That should hopefully be easily fixable. Is termmux using a TextView for the console content or is it using some other kind of view.

What I can say for sure is that if a TextView is being used then brltty has no problem at all knowing where the cursor is. Perhaps, therefore, I should ask a different question. How does termmux not knowing where the cursor is manifest itself?

How to fix this is what is being discussed here,

Of course I'll get around to actually trying it, but I'd still appreciate it if someone could describe for me how termux is rendering the console and also how it's asking where the cursor is.

but it may be helpful if you were able to read the issue from the beginning.

I much prefer to tackle any issue from scratch.

What I'm hoping is that termux is using a standard TextView and that it's simply asking which text is currently selected to know where the cursor is. No text being selected is indicated by the start and the end of the selection being both set to the same offset, which is exactly where the cursor is. Put another way, text not selected is simply the devolved case of the selected text having a length of zero.

Jun 16 '22 20:06 DaveMielke

Could you please explain to me what your goal is?

The short term goal is to fix cursor position of the terminal being assumed by brltty.

The long term goal is to improve accessibility support for termux app and terminal. I don't know what kind of data or features screen readers or blind people would want from a terminal in terms of easier accessibility, so those goals are yet to be decided on. I also don't have a screen reader myself to test stuff out.

You mentioned that brltty doesn't work with multi-line text views. That isn't correct. It works fine with them.

The vertical offset doesn't seem to be correct in logs and is not 0 on first line and changes and something like 13 as per my tests, also gets messed up if I switch back to gmail email send edit screen, I don't know if it works well with the screen reader even with faulty values, or at least what I am assuming are faulty.

So all I need to do is to install the termux app?

You can install termux from https://github.com/termux/termux-app/actions/runs/2509869570 if you want to test the pull request #2840. You need apt-android-7 variants, like termux-app_v0.118.0+bd6347d-apt-android-7-github-debug_universal, unless you know your phone's arch. Don't use apt-android-5 variants, they are only for android 5/6, with legacy/unsupported packages.

That should hopefully be easily fixable. Is termmux using a TextView for the console content or is it using some other kind of view.

Termux uses a custom TerminalView, which is not a TextView or EditText.

https://github.com/termux/termux-app/blob/master/terminal-view/src/main/java/com/termux/view/TerminalView.java

How does termmux not knowing where the cursor is manifest itself?

Termux knows the cursor position. But since its not an EditText, brltty and other accessibility apps don't know the cursor position. Terminal apps/packages/text editors get position with a request for Cursor Position Report (CPR), but android accessibility can't use that obviously.

No text being selected is indicated by the start and the end of the selection being both set to the same offset, which is exactly where the cursor is. Put another way, text not selected is simply the devolved case of the selected text having a length of zero.

Issue with EditText is that its linear. The start and end index is based on position inside the linear text. The terminal is 2D, so cursor position is the row and column. However, we can extract all the text out of the terminal and convert row and column into a start and end index inside the text, but I am not sure if users want that, since users may want actual row and column when using a terminal.

I have opened the pull request #2840 in which termux sends the row and column via AccessibilityNodeInfo.extras to accessibility apps like brltty. You can apply this patch in your app to get the values. I don't know how you will get the row and column to the screen reader though, since they are not start and end index as already explained.

Jun 16 '22 21:06 agnostic-apollo

I may have expressed myself the wrong way around. The real issue is how termux is showing where the cursor is. The right way to do that within a TextView is to set both the selection start and selection end to the same offset - that of the cursor's position.

I also feel the need to raise a cautionary warning: Please don't try to outsmart screen reader apps. Let them do what they do, and help, rather than try to override, them.

All this talk about doing things like setting the content description is unnecessary. If there's a perceived need to do that then that's guaranteed to be the wrong approach. And, yes, if you're trying to use the content description to render the console's text then the whole concept of where the cursor is has been lost. The content description is only ever meant to be a helpful textual description as to what a widget contains.

Let's say you have a graphic of an analog clock face. Then the content description should be set to the text for the time because a screen reader app can't make sense of the graphic. That'd be a proper use of the content description property of a widget. Another would be when a button shows what it does via an image, in which case its content description should be set to a brief textual description of what action the button will perform.

I suppose it's possible that you render the console text with some fancy font, which'd require you to use what roughly would amount to images on a canvas. No screen reader could make any sense of that, of course, which might be why you're trying to then use the content description property. If this is correct, then I'd say that a much better approach would be to have a mode setting which can do it the graphic way but which can also be switched to a TexstView way of doing it.

Jun 16 '22 21:06 DaveMielke

Firstly, I should add that I wasn't the one who implemented setting the entire terminal text to content description. It was before my time.

Secondly, yes I understand that content description should basically only be used to describe the content.

What I want to do is start fresh. What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor. Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView, each line as a separate TextView or each character of each row and columns as a TextView. I discussed that in https://github.com/termux/termux-app/issues/2543#issuecomment-1157768473. But I am not sure what's ideal since I don't have experience with screen readers.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

https://wiki.termux.com/wiki/Touch_Keyboard#Extra_Keys_Row

https://wiki.termux.com/wiki/Touch_Keyboard#Text_Input_View

https://wiki.termux.com/wiki/Terminal_Settings

Also I can't just convert the terminal view itself to TextView since terminals are very complicated and not just text. I can only make it appear as a TextView, possibly with virtual views. I haven't tested that yet though.

Maybe we can bring attention of someone who has worked on console screen reader support, they may be able to give some helpful info.

Jun 16 '22 21:06 agnostic-apollo

[quoted lines by agnostic-apollo on 2022/06/16 at 14:22 -0700]

Brltty isn't assuing anything. It works with each widget as best it can.

The long term goal is to improve accessibility support for termux app and terminal. I don't know what kind of data or features screen readers or blind people would want from a terminal in terms of easier accessibility, so those goals are yet to be decided on.

My challenge to that is that it makes no sense to build on a faulty foundation. It'd be very difficult to convince any screen reader maintainer to want to support some kind of overly complex mechanism which is known to be merely temporary.

You should be using standard OS capabilities as much as possible. If you don't then you'll never be actually improving accessibility. The most would be possibly improving it for, maybe, just one screen reader at a time.

I also don't have a screen reader myself to test stuff out.

Sure you do. Every Android system comes with TalkBack. All you need to do is to enable it.

The vertical offset doesn't seem to be correct in logs and is not 0 on first line and changes and something like 13 as per my tests, also gets messed up if I switch back to gmail email send edit screen, I don't know if it works well with the screen reader even with faulty values, or at least what I am assuming are faulty.

All I can say, here, is that you're wrong. I depend on those logs for brltty support, and can absolutely assure you that the values they report are totally accurate. In fact, I'm even wondereing why you'd dare to conclude that they aren't accurate.

Termux uses a custom TerminalView, which is not a TextView or EditText.

Then I'd suggest that, as mentiond in my earlier message, you have a mode which does use an EditText.

Termux knows the cursor position. But since its not an EditText, brltty and other accessibility apps don't know the cursor position.

If you really want to improve accessibility then, really, you need to cater to the screen reader apps rather than expect ech of them, separately, to cater to you.

Terminal apps/packages/text editors get position with a request for Cursor Position Report (CPR), but android accessibility can't use that obviously.

I highly doubt that's true for all (or even most) such apps. In any event, anyone who supports a screen reader app is looking for compliance with OS standards, not externally imposed conventions.

Issue with EditText is that its linear. The start and end index is based on position inside the linear text. The terminal is 2D, so cursor position is the row and column. However, we can extract all the text out of the terminal and convert row and column into a start and end index inside the text, but I am not sure if users want that, since users may want actual row and column when using a terminal.

Not so. Sure, the content of an EditText is a single string, but that string contains embedded newlines. Brltty, for example, turns it into a multi-line view so that each line can be individually addressed, rendered, edited, etc.

I have opened the pull request #2840 in which termux sends the row and column via AccessibilityNodeInfo.extras to accessibility apps like brltty. You can apply this patch in your app to get the values. I don't know how you will get the row and column to the screen reader though, since they are not start and end index as already explained.

But that'd require each screen reader app to know how to handle your specific, non-standard widget. Why should each separate screen reader app need to have such specialized knowledge? That's an incredibly fragile interface which probably won't interest many (if any). iSmply designing ones own solution and then telling others that they must implement it is, quite frankly, a no go!

Jun 16 '22 21:06 DaveMielke

[quoted lines by agnostic-apollo on 2022/06/16 at 14:43 -0700]

Firstly, I should add that I wasn't the one who implemented setting the entire terminal text to content description. It was before my time.

That was a very bad decision. :-(

Secondly, yes I understand that content description should basically only be used to describe the content.

I'm glad. :-)

What I want to do is start fresh. What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor.

Presenting the console as a single TextView containing all of the text (i.e. none of the per line or per character stuff) would be the way to go. Screen readers have their own ways to navigate text and there's no need to try to be smarter than they are. Speech ones know how best to do it when speech is being used, braille ones know how to do it best for those who use braille, etc.

Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

That makes no difference to a screen reader. Just show the text and where the cursor is.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView,

This is what you should do.

each line as a separate TextView or each character of each row and columns as a TextView.

Both of these would make life much more difficult for a screen reader app.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

That may make a nice user interface, but all any screen reader app will ever care about is what the TextView is actually rendering.

Jun 16 '22 22:06 DaveMielke

Hello,

I am goiong to respond although I am not familiar at all with Android developemnt. In case you don't manage to find out what you need, please let me know and I will ask around (I have skilled people among my colleagues, for instances).

agnostic-apollo (2022/06/16 14:43 -0700):

What I want to do is start fresh.

That would be really cool. Thanks a lot for yoru interest and the effort you seem ready to invest in all this. I hope it will be benefical to the whole app.

What is the best way a terminal can transmit its text (not really its fonts and graphics) to a screen reader via android accessibility and what would make navigating the terminal easier for blind users, specially in respect to cursor.

As Dave explained, I think, the more you use the widgets already provided by the OS or SDK, the likeliest it is things will "just work".

Note that terminals have two modes, one normal mode in which cursor is always at bottom waiting for new commands and another text edit mode used by nano, emacs, etc in which cursor can move around anywhere user wants.

Yes, we (users of braille displays) are aware of that, but it is not really important. Indeed, when using a braille display, we can visit different fragments of what is on the screen, no matter where the cursor is. We can always locate the cursor, but we are not bound to it and can read even those parts of the screen where there is no cursor. This is all implemented by the screen reader and you, as the developer of an application, do not need to care about all this logic.

I can make the TerminalView appear as TextView with virtual views for accessibility services, with the entire terminal as a single TextView, each line as a separate TextView or each character of each row and columns as a TextView. I discussed that in https://github.com/termux/termux-app/issues/2543#issuecomment-1157768473. But I am not sure what's ideal since I don't have experience with screen readers.

What maters to the screen reader is to be able to figure out thecontent of each line in terms of text and perhaps also the attributes (the color). The screen reader also needs to know where the cursor is at any time. I don't know how this has to be implemented in terms of Android widgets, so I can only provide rather high-level requiremnets.

To give you an example, to access the Linux text-console, screen readers typically use /dev/vcsa. That's to show you the kind of information which is needed.

Btw, by extra keys, it means the toolbar shown at the bottom in the terminal, with keys like HOME, and arrow keys, etc. It can be configured with custom keys via ~/.termux/termux.properties file, will require running termux-reload-settings for changes to take effect. If you swipe left on it, it will open an edit text that can be used to send text to terminal.

I think we will need to figure out together whether these functionalities would make sense and be useful for a blind user using termux from a braille display and a keyboard. My guess is that in such situation we would already be able to do HOME, END etc., without the need to interact with the buttons, but I may be wrong and there may be other functions which are not reachable from the keyboard.

In any case, my asusmption is that this tool bar is not really a priority and that it's best to start by focusing on the terminal itself, at least as a first step.

I hope this helps and am willing to provide as much support as required. I am also available for direct exchanges through an instant messaging system or for Audio/Video calls.

Jun 24 '22 15:06 shindere

I'll get back soon, got busy with other stuff. I implemented cursor positioning based on if both start and end selection indexes are same as suggested and as used by brltty. Need to do more testing to ensure everything is right.

Terminals are word wrapped as well, so standard selection index does not apply and each word wrap is a separate line.

Jun 24 '22 15:06 agnostic-apollo

No pressure, no hurry.

I just wanted to make sure you know there is interest in your wokr and will to support.

At the moment I din't understand what you menat about word wrapping and text selection but if I were you I woulnd't lose time explaining because I suspect everything will become clear when I am testing and have real questions.

Jun 24 '22 15:06 shindere

[quoted lines by agnostic-apollo on 2022/06/24 at 08:13 -0700]

I implemented cursor positioning based on if both start and end selection indexes are same as suggested and as used by brltty. Need to do more testing to ensure everything is right.

Note that brltty also supports selected text. So, if text is selected (which, on Android, also means that there's then no cursor) just set the start/end of the selection as needed.

Terminals are word wrapped as well, so standard selection index does not apply and each word wrap is a separate line.

That shouldn't matter. Just place newline (\n) separators at the right places within the TextView, and set the cursor position (or text selection) according to offsets within that same TextView.

Jun 24 '22 15:06 DaveMielke

Note that brltty also supports selected text. So, if text is selected (which, on Android, also means that there's then no cursor) just set the start/end of the selection as needed.

I want to add text selection support but issue with terminal is that selection can be done anywhere in scroll back history, so if user is at bottom and selection is somewhere above in history, the selection text won't be visible or part of currently viewed text. Or only end cursor will be part of text. Although, could be handled by if both start and end indexes are visible, but user may want to select non visible and visible text as well, so gets complicated.

________________
|               |
|   Scroll back |
|  ^............|
| ........^     |
|_______________|                   
|               |
| Viewable text |
| on screen     |
|               |
|_______________|

That shouldn't matter. Just place newline (\n) separators at the right places within the TextView, and set the cursor position (or text selection) according to offsets within that same TextView.

Not using TextView currently, just setting visible text to AccessibilityNodeInfo.setText() and cursor to AccessibilityNodeInfo.setTextSelection(). The text passed is word wrapped as it looks when a user views a terminal. But since terminal stores cursor as row and column, and not index inside the word wrapped text (after newlines), need to translate row and column to a single index inside the text passed. It seemed to be working fine in brltty logs, just want to ensure there are no wrong offsets or calculations.

Jun 24 '22 15:06 agnostic-apollo

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

Jun 24 '22 16:06 DaveMielke

Dave Mielke (2022/06/24 09:16 -0700):

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

Jun 24 '22 16:06 shindere

[quoted lines by Sébastien Hinderer on 2022/06/24 at 09:18 -0700]

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

Brltty sees the whole TextView, even if all of it doesn't fit on the physical screen. What happens to be visible on the screen isn't relevant to a braille user just like what can fit on a braille display isn't relevant to a sighted user. Another way to put it is that both scroll independently and differently from one another.

Jun 24 '22 16:06 DaveMielke

Regardless of what's actually visible, a TextView always contains all of the text. A user should be able to select a huge chunk of text, which, of course, may not all fit on the screen. That's okay and should be allowed. Just set the selection start/end, and let screen scrolling handle it for sighted users and brltty handle it for braille users.

In context of terminal, I don't think that would be wise. Firstly, user wouldn't know what's actually on the screen if I pass entire terminal text and Talkback will also start reading from top of scroll back history every time instead of visible terminal. Secondly, the terminal by default has 2000 rows and somewhere around 80 columns, so passing 2000*80=160KB on each terminal text update possibly happening many times a second is not really efficient and if accessibility is using binder transactions, which likely it is to transfer data to other apps, then that would trigger TransactionTooLargeException, since max limits are around 100KB, would need to test if it does trigger exception, not sure how browsers handle that. Rows can be increased to 50000, resulting in 4MB data, although likely a blind man may not do that.

When termux shares terminal text, it truncates the data to 100KB to prevent exceptions. I guess, when user goes into selection mode, then can start sending all the text, or after truncation.

https://www.reddit.com/r/tasker/comments/prro8t/autoshare_crashed_when_i_pasted_the_file_path/

Jun 24 '22 16:06 agnostic-apollo

Dave Mielke (2022/06/24 09:30 -0700):

[quoted lines by Sébastien Hinderer on 2022/06/24 at 09:18 -0700]

About scrolling, do we agree that scrolling in brltty should scoll for sighted users, and likewise the other way around?

Brltty sees the whole TextView, even if all of it doesn't fit on the physical screen. What happens to be visible on the screen isn't relevant to a braille user just like what can fit on a braille display isn't relevant to a sighted user. Another way to put it is that both scroll independently and differently from one another.

I'm not sure I agree things always need to be so and can absolutely not be different.

For instance on the Linux text console, when shift-pageup and shift-pagedown were working, what was visible to the visually impaired user was exactly what was on the console and that seemed nice to me.

I realise that when using a braille display one does not always be limited to what there is on the screen of the phone, but I also think there may be cases where synchronisaiton is desirable, typically if a visually impaired user and a sighted user would like to work together.

Jun 24 '22 16:06 shindere

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

Jun 24 '22 16:06 shindere

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

You may be able to do that in landscape mode, not in portrait mode since rows would be greater than columns. I guess terminal margin could be adjusted with terminal-margin-horizontal and terminal-margin-vertical properties and/or terminal zoomed to appropriate font size with pinch and zoom of fingers to get that in portrait mode too.

Jun 24 '22 16:06 agnostic-apollo

agnostic-apollo (2022/06/24 09:54 -0700):

If using termux without a sighted user, I'd definitely like it if I could have the very same terminal than on Linux in terms of dimensions, i.e. 25 rows of 80 columns.

You may be able to do that in landscape mode, not in portrait mode since rows would be greater than columns. I guess terminal margin could be adjusted with terminal-margin-horizontal and terminal-margin-vertical properties and terminal zoomed to appropriate font size with pinch and zoom of fingers to get that in portrait mode too.

Thanks. Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else. The device is called a BrailleSens 6, from HIMS.

Jun 24 '22 17:06 shindere

[quoted lines by Sébastien Hinderer on 2022/06/24 at 10:12 -0700]

Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else.

And may I dare to add that accessibility should never be added with just one use case in mind.

Jun 24 '22 17:06 DaveMielke

Dave Mielke (2022/06/24 10:22 -0700):

[quoted lines by Sébastien Hinderer on 2022/06/24 at 10:12 -0700]

Just to make it clear, on the device where I'd like to use termux, there is no usual screen. Just one LCD screen that shows to a sighted user what is on the braille display, nothing else.

And may I dare to add that accessibility should never be added with just one use case in mind.

For sure. I just mentionned mine as one among others, definitely not as the only use-case.

Jun 24 '22 17:06 shindere

The device is called a BrailleSens 6, from HIMS.

You would still need to configure termux as per your rows and columns requirements so that data is sent to braille device accordingly. In future, shell APIs are planned to configure terminal font size, margins, rows, etc.

Jun 24 '22 17:06 agnostic-apollo

termux-app termux-app copied to clipboard

Let screen readers catch cursor position

termux-app
termux-app copied to clipboard