ghidra
ghidra copied to clipboard
NationalSecurityAgency
Make listing and decompiler areas accessible for screen reader users
When trying to use Ghidra with the NVDA screen reader, a lot of the UI is accessible, however, the likely most important areas (the machine code and the decompiler listing displays) are reported only as panel without any way to access their content. That's somewhat frustrating, because as a visually impaired developer you can not rename variables and do all the useful stuff. I have no idea how complex the needed changes would be (i've never written such complex Java GUI app), so even i'd much prefer to make this a bug report, i'm reporting this as a feature request instead.
The Listing and the Decompiler (along with some other widgets, like the Byte Viewer and the Function Graph) are custom widgets whose rendering we control directly. I'm assuming standard Java widgets are known to the screen reader you mentioned. We will have to see how we can update our custom rendered components in order to provide data to the screen reader.
That's true. A quick Google yielded this document from Oracle for starters: https://docs.oracle.com/javase/tutorial/uiswing/misc/access.html Maybe it'll be useful that the source code is available, i assume that these widgets will not be as straightforward, but thansk for the response anyway.
I did some brief research on this. I was hoping for a quick 80% solution, but it appears that it may not be that easy. Java has a layer of support built-in for accessibility, but it seems as though it was never fully implemented. Compounding the issue is that each platform's reader tools vary wildly in how well they use Java's API.
In theory, our custom widgets should be able to tap into Java's existing API without too much effort. We won't know exactly how much work it will actually take until we can get some tools to test with on each of the platforms. My gut feeling is that we can still provide a reasonable improvement without too much development effort.
I thought this over carefully. In my opinion the AccessibleText interface from javax.accessibility needs to be implemented for the custom-rendered widgets, and probably also the AccessibleSelection interface for the byte viewer and the function graph widgets. These are mature, well-documented interfaces that exist since at least from Java 7 (maybe earlier). Findig their doc is trivial by their names. After reading through the doc for these interfaces, it seems a moderate amount of work to implement these for the given widgets. And then screen readers would get the neccessary info about these custom widgets for their purposes and that alone would make them accessible. That's how this works. Testing on Windows is best done with the free and open source NVDA screen reader obtainable from www.nvaccess.org orr fromtheir github page. On Linux Orca is the only screen reader that is equipped with Java accessibility support. Best test bed is Ubuntu Mate, just press Alt-Super-s to launch Orca. MacOs has VoiceOver screen reader built in, but I don't know about its Java interaction capabilities, as I have never used a Mac. I hope this info helps much and puts this important issue back on track toward solution.
One more important thing: in order to implement the accessibility interfaces mentioned in the previous comment for the custom widgets, first the Accessible interface from javax.accessibility needs to be implemented for them, if not implemented already in some way. That interface has one method: getAccessibleContext which should return an AccessibleContext class. That class is the "root hub" for all things connected to accessibility for that widget. That class contains basic info about the widgets for screen readers, like name, description, accessible parent and children etc. That class also has additional methods through which such objects can be returned that implement the mentioned accessibility interfaces. E.g. for the listing and decompiler area widgets, the AccessibleContext class for them should contain the getAccessibleText method that should return such an object that implements the AccessibleText interface and can expose the caret position and text itself in those areas together with its visual representation (text attributes and location on screen) through that interface to the screen reader.
I am willing to work on this very much. I have some very very early evolving prototype code already, which I refine as I explore docking.widgets.fieldpanel internal logic step by step. However I face very tough and nasty crashes of NVDA screen reader on the code browser main ui after certain interactions with it (on original, unmodified Ghidra). These hinder my work very much and if not resolved somehow will make my testing (once I get there) practically impossible. I reported this crash thing to NVDA devs in detail here: https://github.com/nvaccess/issues/12045 @tyrylu please read through the linked issue. Have you experienced anything like that? Can you reproduce the crash with the given steps? If yes please report your experiences at the linked issue in comment. That may help NVDA devs much. Thanks.
I correct the issue link I have just posted: https://github.com/nvaccess/nvda/issues/12045 I typed it by hand and omitted something. Sorry.
Are there any updates on this issue? The latest Ghidra seems to somewhat work with these screens, except the text fields don't seem to read. They just say blank, or sometimes "Line 1: No function".
There has been work done to add screen reader support to Ghidra for theL isting View, Decompiler View, and ByteViewer View. From what you wrote, I can't tell if it is mostly working and there are a few places where is isn't or it mostly isn't working at all. So what view are you talking about? And what do you mean specifially by "text fields"?
Sorry, I should've been clearer. It certainly works better than anything else I've tried, but some stuff doesn't, mainly parts of the decompiler and listing views. When I say text field, I mean an edit field (something you can typically type in). I'm actually entirely uncertain how this looks visually, but my screen reader, NVDA, sees two multiline text fields in this view, one for the listing and one for the decompiler. Perhaps these have the wrong roles and that's why they don't work? In either case, it's mainly these that I have problems with. The rest of Ghidra can be made to work using a combination of hierarchical navigation keys, or with OCR, but I just can't get either of these areas to display anything useful. I can access all the buttons and everything, I just can't access the info. I'm assuming it's stored in these edit fields.
Do you have a program open? Have you imported a program? Until you do, both the Listing and Decompiler are empty. I ask because before you mentioned the Decompiler was reporting "No Function" which is what it displays when there is no function selected in the listing (or no program is open). Last summer, I tried adding support for screen readers for these Ghidra custom components. I don't know how successful I was, because I have gotten no feedback. I don't think anyone else besides me has even tried it. I used NVDA to do my development, but the support tools and documentation I could find were not very robust and our development environment are severely restricted. Anyway, at least for me, the screen reader provides info on each field in the listing and decompiler as I move the mouse over them or navigate to them using the cursor or tab. So I guess I expect it to report some info for each field. I thought the biggest issue would be if I was providing too much info or too little.
Also, keep in mind, that neither the listing or decompiler views have any input fields. They are display only, unless you invoke some action using keybindings or popup menu that pops up an input dialog.
If it is the case that you don't even have a program open, perhaps i need to add code to specifically handle that to say something like "No program open" instead of just saying "blank" or nothing if there is no program open.
I have gladly read that initial screen reader support was put in place for the listing and decompiler views and the byte viewer, so I would have definitely wanted to test this with NVDA 2023.3 and Ghidra 10.4 (the latest version that has this feature), but the only machine I currently have that meets all Ghidra system requirements thought that it simply wouldn't turn on, so currently boom, I am out of luck with this :(
2023-12-05 18:02 GMT+01:00, ghidragon @.***>:
If it is the case that you don't even have a program open, perhaps i need to add code to specifically handle that to say something like "No program open" instead of just saying "blank" or nothing if there is no program open.
-- Reply to this email directly or view it on GitHub: https://github.com/NationalSecurityAgency/ghidra/issues/726#issuecomment-1841236041 You are receiving this because you commented.
Message ID: @.***>
Are you saying that Ghidra does not run? Or that Ghidra runs, but the screen reader is not working with Ghidra?
No, I have a program open. They're read-only text fields, yeah. But the contents just don't read. Pressing the arrow keys in both of them just says blank, no matter what I do. In addition, I can't tab around at all.
This all seems to be working for me.
-
Can you post a screenshot of the tool with the UI widget that contains the cursor when you are trying this?
-
Try this:
- Place your cursor in the Listing view on an assembly instruction
- Press the arrow key
- The cursor should move to the next letter
- Regarding the term 'text field', is the screen reader using this term? Or, are you using this term generically to mean a UI widget that contains text?
I opened an executable in Ghidra, tabbed until I found the code browser button, and hit it. I then was landed on a tree view that only contains the node of "data types". If I expand it I get a few random things, but nothing super important. If I tab I then get a filter text field, and eventually land on the listing view. My screen reader says "Listing edit blank". It calls this an edit field, the same way it calls the code browser that, although it says "read-only" as well in that case. This is what I mean when I say text field. Attempting to navigate around the listing view at all just says "blank", and I'm also unable to tab out of the text field to other controls once I enter it.
I just tried running the screen reader in a different environment (on a mac) and I'm seeing the behavior you are describing. I will investigate.
Ok. So when I run this on the amazon ec2 instance where I did the work, the screen reader seems to work. Every time I either move the mouse over something, it reads it to me. Or if I tab or arrow around in the listing view it reads to me. For example, if I'm on a field and press "right arrow", it will read just the letter if I'm on the same field. But if that causes it to move to the next field, it reads the entire field text.
My ec2 instance is running a server version of Windows and I am using NVDA 2022.4. But when I run this on my personal Mac system, I get behavior very similar to what you describe. When I first click on the listing, I get a full reading of whatever field I clicked on, but then moving with the cursor I get nothing at all. Clicking with the mouse, I get something, but it is wrong and says something like "field is blank" and that I'm on a text field, similar to what you reported. But if I give keyboard focus to some other component and then give focus back, it again reads correctly one time and then is broken again as long as I remain in the listing view.
I think my next step is to update the NVDA version and see if it still works for me. I suppose it could also be related to the version of Java I'm using. Anyway, I will continue to investigate this.
I have since tried this on a windows desktop computer running Windows Home and NVDA 2023.3 and it worked as expected. I also asked someone with a windows laptop (also running Windows Home) to test it out. At first, the NVDA was not working at all with Ghidra. But after fiddling with it a bit (and maybe there a reboot in there somewhere, not sure), it suddenly started working after checking some setting that made the caps-lock key be the NVDA key. But a lot of settings were tinkered with, so I don't think it really had anything to do with the NVDA key being caps-lock. Anyway, the bottom line is that it did eventually work.
Reading back through your posts, I really think you don't have a program open in the code browser tool. Having a program open in the tool is not the same as having imported a program in the project window. When you run a tool, you have to explicitly open a program, either by dragging a program onto the tool or using the "open program" menu item. If I run a tool and don't open a program, I get exactly the behavior you describe. The listing window, just keeps repeating "Blank" as you press the arrow keys. There are several other places you can navigate to verify that a program is open. For example ,the "program tree" window will read "no program" if a program is not open or node name will be the name of the program. Also, if a program is open, the data types tree will have three children under the root instead of just two and one of them will be the name of the open program. Also, if a program is open, the File->close program menu will contain the name of the open program and not just say "close program".
It would appear that you're right! I still have a good amount of accessibility feedback for improvements, but this works, and I'll open an issue for those. Thanks so much!
Great! I know there are lots of issues, especially regarding keyboard only navigation. That is an area we hope to address soon. But any feedback on the screen reader support specifically would be very helpful.