liblouis
Documentation Enhancements (work on the manual in order to lower the barrier)
I'm sorry for the very general title but the aim here is to discuss and track some of the improvements for the documentation. I know there are other issues open about the documentation so I might try and combine some of those in to this but for now I'll start here. Over the next while I will be going through and trying to improve some or all of the documentation in order to lower the barrier of entry for adding tables, updating the documentation, and maybe contributing to the source code. If there are any things that you would like to see in particular please let me know and I will try my best to include them. I'm open to any suggestions whether it's about the technical side of the documentation such as the language used or just things you would like explained better.
Oh hey, I was just thinking about this. I struggled a lot with the documentation when I was trying to figure out how to get this program running. I would be happy to help write documentation if it would be helpful to have an outsider perspective.
Here is what I wished for, as a sighted software engineer who is new to braille, when I wanted to add a new language to LibLouis:
- If I want to add a table for a new language to LibLouis, what files do I need to add, and what files do I need to change?
It seems like the answer is that you need to create two new files:
- The braille table: tables/your-language.utb
- The test file: tests/braille-specs/your-language.yaml
And you need to add your language to the lists in:
- tests/Makefile.am
- tests/braille-specs/Makefile.am
- tables/Makefile.am
- extra/generate-display-names/display-names
- .github/workflows/fuzzing.yml
- AUTHORS
- NEWS
- How to add and run the tests?
a) What is the recommended way to write the test file?
What I did was, I typed up my test cases, and then letter by letter, I looked them up in my braille table to figure out what the dots should be, then I googled "braille dots 136" or whatever, and found the unicode character for those braille dots, and then pasted that unicode character into my test file. I thought that surely there must be a better way to do it, but I didn't know what that better way was.
b) How do you actually run the tests?
What I eventually figured out was that I had to run:
export CPPFLAGS="-I/opt/homebrew/Cellar/libyaml/0.2.5/include $CPPFLAGS"
export LDFLAGS="-L/opt/homebrew/Cellar/libyaml/0.2.5/lib $LDFLAGS"
./configure
make
sudo make install
make check
But even after I figured that out, I wasn't really sure which of those lines did what. I think make compiles LibLouis, and sudo make install installs it as an executable program on my machine, and then make check runs the tests, but those are just educated guesses - it would be great to have documentation that explains what's going on. And I also wasn't sure which of those commands I needed to re-run after I had made changes to my braille table or to my test file. Did I need do re-run make and sudo make install after each change to my braille table? Did i only need to run make check again if I had only changed my test file?
- Once I have my braille table, how do I use it to transform a text file into braille?
I eventually figured out that I needed to run lou_translate --forward unicode.dis,yi.utb < path/to/file.txt and then copy-paste the output (which was unicode pictures of braille dots) from stdout into a text file, and then from there I used Harris Mowbray's website to convert the unicode braille dots to ascii braille, and copy-pasted that into a new text file, and then I saved that file with a .brf file extension, and that was how I eventually figured out how to turn a text file into a file that could be read on a braille device. There might be a better way to do this that could be documented.
That's what I needed to know. If I were just a language expert but not a software engineer, I would probably also need to know: how to make a fork of the repo, how to make a branch, how to open a pull request, how to write tests, and how to get the program running on your machine.
Another question I have for the FAQ list is this:
What is the difference between a display table and a translation table? What are the differences between the different l file extensions used in the language tables? How do I decide which file extension I need for my table?
It seems like the answer is that you need to create two new files:
- The braille table: tables/your-language.utb
- The test file: tests/braille-specs/your-language.yaml
And you need to add your language to the lists in:
- tests/Makefile.am
- tests/braille-specs/Makefile.am
- tables/Makefile.am
- extra/generate-display-names/display-names
- .github/workflows/fuzzing.yml
- AUTHORS
- NEWS
The essential things are of course the table and the YAML tests (your-language.utb and your-language.yaml). As a second step you want to add the table and the test to the build system, so that the tests are run and the table will be packaged in the distribution. This entails
-
tables/Makefile.am -
tests/Makefile.am -
tests/braille-specs/Makefile.am
The display-names file is used to check the meta data of the table. For exact details you'd have to ask @bertfrees
The fuzzing.yml is just an annoying side effect of how the fuzzing tests work. This file is updated automatically if you change the Makefiles. This is not really something that a table author would have to worry about too much.
We put names in the AUTHORS file to acknowledge contributions. It is not mandatory but I think it does nicely acknowledge the work that people have put into the project.
Lastly the NEWS file is used to create an announcement for the next release. Not every change will be mentioned in there but new tables are certainly interesting for people to know. Due to the way it is set up we often get merge conflicts, so I often just like to sort have a summary of the changes, e.g. in a PR and then I put this into the NEWS file myself
a) What is the recommended way to write the test file?
What I did was, I typed up my test cases, and then letter by letter, I looked them up in my braille table to figure out what the dots should be, then I googled "braille dots 136" or whatever, and found the unicode character for those braille dots, and then pasted that unicode character into my test file. I thought that surely there must be a better way to do it, but I didn't know what that better way was.
I think this highly depends on the person. For a sighted person it is quite hard to "read" the braille, so it might be better to have the tests written using some form of ASCII braille. You are not bound to use the unicode.dis display table. You can use any display table you like in your tests. There are some YAML tests that use ASCII braille, e.g. de-comp6.yaml.
If you are an Emacs user you can use a special input method that allows you to enter the braille by entering the dot numbers
I cannot speak for braille readers, but I presume that for them entering braille unicode comes quite natural.
b) How do you actually run the tests?
What I eventually figured out was that I had to run:
export CPPFLAGS="-I/opt/homebrew/Cellar/libyaml/0.2.5/include $CPPFLAGS" export LDFLAGS="-L/opt/homebrew/Cellar/libyaml/0.2.5/lib $LDFLAGS" ./configure make sudo make install make checkBut even after I figured that out, I wasn't really sure which of those lines did what. I think make compiles LibLouis, and sudo make install installs it as an executable program on my machine, and then make check runs the tests, but those are just educated guesses - it would be great to have documentation that explains what's going on. And I also wasn't sure which of those commands I needed to re-run after I had made changes to my braille table or to my test file. Did I need do re-run make and sudo make install after each change to my braille table? Did i only need to run make check again if I had only changed my test file?
If you run from a git checkout all that should really be needed is
./autogen.sh
./configure
make check
Since you are only interested in your table you could also just type
./autogen.sh
./configure
make check TESTS='braille-specs/your-table.yaml'
The configure script should find your libyaml installation via pkgconfig. If that doesn't work I guess you will need to set the CPPFLAGS and LDFLAGS manually.
The make install step should not really be needed just for testing. It should use the tools it built locally in ./tools, and test the tables in the git checkout. At least that is what it does under Linux.
If you change the table only, then a simple make check TESTS='braille-specs/your-table.yaml' should do. This should give you a summary of the test run. If the tests fail you can consult tests/test-suite.log for more details on which tests fail and a diff of the expected and actual results.
- Once I have my braille table, how do I use it to transform a text file into braille?
I eventually figured out that I needed to run
lou_translate --forward unicode.dis,yi.utb < path/to/file.txtand then copy-paste the output (which was unicode pictures of braille dots) from stdout into a text file, and then from there I used Harris Mowbray's website to convert the unicode braille dots to ascii braille, and copy-pasted that into a new text file, and then I saved that file with a .brf file extension, and that was how I eventually figured out how to turn a text file into a file that could be read on a braille device. There might be a better way to do this that could be documented.
Couldn't you just use lou_translate --forward text_nabcc.dis,yi.utb < path/to/file.txt > path/to/file.brf? There is probably a way to do this better with file2brl.
For a while I toyed with the idea of using Pandoc for document conversion and using a Pandoc Lua Filter to translate the text to braille and let Pandoc worry about the formatting. IMHO this idea has a lot of potential, but I seem to remember that there was a problem with my lua bindings for liblouis, so I never finished my experiments. This idea deserves to be resurrected.
That's what I needed to know. If I were just a language expert but not a software engineer, I would probably also need to know: how to make a fork of the repo, how to make a branch, how to open a pull request, how to write tests, and how to get the program running on your machine.
This kind of stuff is probably best documented by referring to the official git(hub) documentation. Maybe with an eye towards braille display users. There is some (way too wordy) material in the wiki.
@egli @abbyhowell thanks so much for your comments. I just got back from a week off and before I went away I was having problems building Liblouis. I think it's because I cloned the repo on my computer which is running windows but I was trying to build inside of a dev container and the Dockerfile kept failing to build even when I was trying to use that instead of a fresh Debian install but those are some great ideas for the documentation and would definitely make things easier and clearer. I'm hoping to get some time to figure out why things aren't building during the week and then I can start making some actual progress and create a draft PR to share with you all.
@egli @bertfrees how would you feel about splitting up the documentation in to multiple pages. I personally think it would be better if each page was to help you perform a certain task e.g. one of the pages would be getting-started with creating a table and another would be about using the helper functions etc. Let me know what you think
Just to be clear (and to maybe improve the general title of the issue), the goal of this issue is to work on the "Liblouis User’s and Programmer’s Manual", in order to lower the barrier, right?
I like the idea of a multi-page manual. Perhaps we can have the two: multi-page and single-page.
To get multi-page I think we just need to drop --no-split from the makeinfo command. And also the --no-headers.
By the way I really like the style of the documentation of GNU software. See e.g. the documentation of bash or Emacs. You can choose between multi-page HTML, single-page HTML, and some other formats. The source is texinfo, just like Liblouis. To get the same style I think it's just a matter of applying this CSS style sheet.
It also seems possible to use org-mode for the source, and export it to texinfo, using org-texinfo-export-to-texinfo. This is what e.g. Magit does.
@bertfrees yes exactly and that's a great idea. I'll take a look at those. You are correct about splitting it up we just need to modify the make file.
@abbyhowell , when I test tables, I just use the lou_checkyaml program. It's a lot simpler than running the UNIX commands you mention. lou_checkyaml testfile.yaml
One thing that I'd find useful in the documentation is more examples of how to use the various opcodes, and how to use them in combination. There is a gap, for example, between the simple examples of say, match, found in the documentation, and then the actual rules you find in tables. So suggest more examples. Also, it would be helpful to have some recommendations for "idioms". For example if I wanted a rule that matched against the beginning of a word, I could use a variety of different classes (e.g. a positive space or beginning of file etc, or a negative "not an alphabetic").
@paulGeoghegan I was wondering if you have already looked into the multi-page option? If not, I'd like to look into it now.
@bertfrees go ahead. I honestly haven't had nearly as much time for this as I had hoped and I have wasted a lot of it on trying to build Liblouis. I'm still going to try and contribute to this as much as I can though.