"Big" ideas for future development

[NSoiffer]

Collecting Ideas for future development...

Synchronized highlighting

When a key command is used to speak an expression...

• Could use speech API, but only get word callback boundaries
• Could use google speech API with fall backs SRE integration
• How does sync highlighting work with SRE?
• Can we carve out some part of SRE because SRE is big?
• Can we pass maston to SRE with bookmark indicators Pass info to SRE/speech engine
• Modern speech engines support SSML and bookmarks (callbacks happen when bookmark is reached); speech continues
• Web SpeechAPI only does bookmarks at word boundaries
• Modern speech engines allow an arbitrary string for a book mark. This could be ID attr value or possibly some augmented value to reference a character or character range (e.g., 123 or sin)

Error warning/correction and expression prediction

We can gather data on likely expressions, perhaps using n-grams or machine learning to either say something is likely/unlikely. This can be used to flag potential mistakes (e.g., a missing coefficient in a polynomial or an unlikely combination of variables (e.g., 2sd).

Alternatively it can be used to give potential endings such as offering bx + c after a user types ax^2.

Semantic editing

We should find a non-obtrusive means to allow uses to enter and/or supplement information in the editor so that the editor has a better idea of what the notation means. This might influence speech or computation.

One idea that is already implemented is a semantic palette where a user chooses from among alternative meanings for a given notation. Other possibilities are:

• keyboard equivalents (perhaps LaTeX style commands such as \abs{x}
• providing a subject area and inferring meaning based on the subject area. E.g, P(A|B) is 'probability of A given B' if we know the subject involves probability
• type inference -- if the user has given some information about a variable or operator, more information about other variables and operators can be inferred. This includes knowing when something is a unit, it likely means other quantities also must involve units so things like m as a trailing part of a term likely means 'meter' rather than a variable.
• using a jagged underline when the editor has to use a heuristic that has a higher (for some value of "higher") chance of being wrong. This might include some cases of function call vs multiplication (e.g, h(a+b) where h might be a height function or it might be the height of an object.

Elementary Math

Math editors that support kids in grades 1 - 5 are few and far between. Accessible ones probably don't exist. Elementary math consists of stacks of digits, horizontal lines, along with +, -, and cross outs for borrows & carries. Long division adds a little decoration but is otherwise similar.

These are difficult to enter using the standard WYSIWYG paradigm. They likely need to be templated. Probably legal characters should be restricted. Also, only some notations should be allowed. Entering numbers may even be better done right-to-left, but I'm not sure of that.

The implementation is not that difficult because the notation maps to tables where each entry in the table is a digit or maybe an operator sign. Borders are used for lines.

An extension of long division to synthetic division is useful because that is taught in Algebra II. Instead of digits in the columns, powers of 'x' along with the coefficient go in each column. Blanks replace 0's when the coefficient is 0.

Touch Screen Support

• Gestures
• Cross outs
• Drag and drop

Speech Input

Math and commands (e.g, delete, select numerator, ...)

Handwriting Input

Handwriting input exists for math and is pretty good. However, it suffers from at least two drawbacks in the current implementations:

• it is static -- if you want to insert something, you can't open up space. Conversely, if you delete something, it doesn't close the space up
• it lives in its own world. You can't point somewhere and speak some input or command, nor can you enter or delete anything from the keyboard.

[Kreijstal]

if you ever need an already done project, it might be possible to copy from https://webdemo.myscript.com/views/math/index.html#

[sjcjoosten]

I'd love to be able to easily determine the set of symbols that are available. For example, if I have a modal logic exercise question where users need to fill in the blank, I'd like it for the answer to be typed with \square and \diamond, and not have users see \sqrt when they type '\sq'. Similarly, there is no need for the \sqrt to show up on the virtual keyboard.

[NSoiffer]

I'm at an accessibility conference and speech input for math has come up a few times. Is anyone working on adding speech input to mathlive?

[arnog]

Closing. If there is interest in specific items, I would suggest opening separate issues as Feature requests.