Real-time in-browser translation for all books

[mekarpeles]

Problem

How can we make our books more useful and accessible to a global audience? Many books are published in a limited set of languages, making them entirely inaccessible to broad classes of international readers, similar to how many print books are inaccessible to the blind and dyslexic. This constitutes a major issue with equitability.

Solution

One possible remedy is to integrate assistive technologies within our book reader which enable the reader to translate individual book pages on-the-fly into a language that is useful to them, similar to the affordance many web browsers already provide to translate many web pages. This affordance may remove barriers and enable audiences who do not know a book’s language to still access its content.

See: https://blog.mozilla.org/en/mozilla/local-translation-add-on-project-bergamot/

Precedent

Many web browsers already offer a “translate this page” feature that works with many plaintext websites, automatically detect when a web page’s text is in a foreign language, and offer to translate the page.

Constraints

Because the Internet Archive bookreader relies on images and hidden text layers for its reading experience, many web browsers aren’t configured correctly to apply their standard “translate this page” feature.

Today, a patron may manually right click on the book page and select “translate this page”, but the translation gets applied to the hidden text layer but is not shown to the reader. There's no easy way via javascript to trigger a translation call or customize the translation behavior because most browsers don't offer a translate API which allows one to augment how translation occurs. By using "translate this page" and then manually setting the background and color of the hidden text layer using css in developer tools, we can demonstrate that the translation is indeed working and being applied:

Because of this first limitation, the reader my instead elect to highlight a section of the book and use the browser’s translate function to popup a translate box, but this manual process is significantly less convenient than the browser’s standard feature to translate the page.

Since the browser's default "translate this page" feature doesn't work well with the bookreader's unique implementation and the process to manually select and translate a region is challenging to discover and use, a third option is to consider building a customizable translation solution capable of accommodating the book reader’s unique implementation. One option is to use Google Translate APIs but this is a burdensome, unsustainable expense for a nonprofit project at our scale.

Fortuitously, recently Mozilla announced that it had ported Project Bergamot -- a set of neural machine translation tools -- into this web assembly implementation, which could be used to prototype a client-side translate on-the-fly feature (see mozilla's demo) for the Internet Archive’s books:

https://blog.mozilla.org/en/mozilla/local-translation-add-on-project-bergamot

Proposal

Builds off of the text layer work of https://blog.openlibrary.org/2020/08/29/giacomo-cignoni-my-internship-at-the-internet-archive and use mozilla's web assembly implementation of the Project Bergamot neural translation models to see if we can augment our book reader by overlaying uneditable translations, on a per-page basis, to enable books to be read in other languages.

Here's a depiction of how this feature might look:

Steps

1. Port the mozilla demo into a minimal stand-alone example that can be run locally, offline, with downloadable models https://github.com/mekarpeles/ia-br-translate-experiment
2. Integrate this minimal https://github.com/mekarpeles/ia-br-translate-experiment example into the bookreader https://github.com/internetarchive/bookreader/tree/auto-translate which can be demo'd on http://localhost:8000/BookReaderDemo/demo-internetarchive.html?ocaid=adventureofsherl0000unse
3. Port this bookreader example to use netlify so it can be tested in the browser with an Internet Archive public domain book
4. Polish this example so that it runs well in the bookreader and add a feedback button to e.g. get ⭐ or comments about quality of translations.

Implementation Notes

https://docs.google.com/document/d/1jUlOCoL5LTkmj-qn7-6Nf8aul1pTlhHHkJydIidWIfs/edit#heading=h.ma1056kmf3d8

[bicolino34]

I think, human-translated books would still be more preferable to people. We could improve the number of modern book editions by adding unauthorized fan-translations. I've asked about adding them in the past, but the stance then was not to add even just the metadata record due to potential copyright infringement.

[mekarpeles]

Current State

Next time on [mek to insert]:

• Split index.js into Demo related code (move to demo.html)
• Change index.js to expose an API (see 2024-08-29)
• npm init to create a new npm module at @internetarchive/web-translate
• Try importing in BookReader and work out kinks
• Test/experiment with unpkgify (use import.meta)

Design

type WebTranslate {
    async init({ workerPath: Optional<str>, modelBaseUrl: str }): Promise<bolean>;
    async languages(): Array<{ code: str, name: string }>; // (Or do they need to be from-to pairs??)
    async translate({from: str, to: str, text: str, }): Promise<string>
}

Logs

2024-08-29

Ideal interface:

<script src="https://unpkgify.com/mozilla-translator">
<script>
    MDNTranslate
    MDNTranslate.loadModule(from, to);
    MDNTranslate.languages();
    const translated = await MDNTranslate.translate(string, from, to);
</script>

<script type="module">
    import MDNTranslate from 'https://unpkgify.com/mdn-translate?esm';
    MDNTranslate...
</script>

ia-moz-translate

Strategy for an easy to use general library:

1. Publish to npm

Strategy for BookReader:

1. Publish to npm prefixed eg web-translate An experimental, convenient wrapper of the bergamont mozilla translate API & models, hosted via the Internet Archive

   • js/index.js (slim, just the API)
   • js/worker.js
   • js/bergamot-translator-worker.js
   • js/bergamot-translator-worker.wasm

• demo/ index.html w/ demo.js + demo.css

Use import.meta API to get the path of the current javascript (eg. https://unpkgify or not)

[mekarpeles]

See this prototype: https://github.com/mekarpeles/ia-br-translate-experiment

Run ./setup.sh (assuming you have wget installed) and then run with python3 -m http.server 8000

[mekarpeles]

Mozilla released a web asm Translation Model that can be run within the browser, and this means web applications, like our Book Reader, can provide on-the-fly text translations across languages.

We have a prototype here which includes working code. It automatically fetches the models from archive.org if the user doesn't have them yet, or uses the cached version on disk: https://github.com/mekarpeles/ia-br-translate-experiment

This project builds off of the work of https://blog.openlibrary.org/2020/08/29/giacomo-cignoni-my-internship-at-the-internet-archive/

The desired outcome is a UI like this:

The big unfinished piece of this project is moving this code inside the bookreader, overlaying the translated vectorized text, and getting it to work with read aloud. The way Book Reader works is that the patron is shown images.

So in order to do things like text selection, search inside, or other types of "text alignment", we render invisible svg text above the image. That's how one can highlight and copy a section of e.g. public domain or open access books. In this case, we would use this svg text layer to send the untranslated text in to this model... And replace the svg layer with the new translated text... And then make it visible (with a beige background) to hide the original images behind it.

[bicolino34]

How does it perform on Public Domain books, which usually don't have the best OCR layer?

[mekarpeles]

@bicolino34 Not well, as one would expect! :)

[schu96]

@mekarpeles I've managed to preserve the hidden text layer and have gotten to the point where I can intercept the TTSEngine's chunk reading to read one page of translated text and change the TTS voice to automatically match the language that it is being translated to.

Are we looking to completely mimic the text to speech function for the translated text with the exception of text highlighting? The current major issue that I see with my implementation is that there are no sentence chunks like the original implementation, which can be an issue if a user wishes to pause and resume the text to speech reader.

[mekarpeles]

@schu96 this:

• https://github.com/internetarchive/bookreader/issues/1316

Is an interesting proposal, as it may allow us to create a better overall TTS experience (irrespective of translations) and also make it so we can translate the entire page and update the text layer prior to read aloud. 🤔

[Alok-2005]

@mekarpeles is this issue still open ?

[mekarpeles]

@Alok-2005 yes it's still open, @schu96 is currently driving an effort forward, though it's possible she or @cdrini may have thoughts on how others could contribute?

One thing I don't believe @schu96 is working on but could be useful is finding a way to package the worker stuff as an importable npm module or package that any project could easily import and use with the archive.org item's models... Though I think this is lower priority on the list.

[mekarpeles]

@schu96 we may want to update this issue or to link a google doc that describes what approach we plan to take with respect to the difference in implementation between translating the entire page v. TTS in translate mode.

[Aditya-132]

may i work on this issue

[mekarpeles]

@Aditya-132 someone's currently assigned to this issue. Check our other fellowship level opportunities if you'd like an available issue similar to this one

https://github.com/internetarchive/openlibrary/issues?q=state%3Aopen%20label%3A%22Fellowship%20Opportunity%22%20no%3Aassignee

[mekarpeles]

Here is the design @cdrini and I converged on

OR

If you update original language, this will update in both sections. If you click the non-selected language (e.g. French), it will make it selected. If you click the down arrow, it will open a select box.

[mekarpeles]

Checklist

• [ ] Better Loading Strategy: e.g. Show loading spinner until page fully translated
• [ ] Cache Translations → (page, paragraph) + show/hide translation layers(s) by language pairs (e.g. “sp-en”)
• [ ] Load/separate additional pages/translations in background
• [ ] Update configurePageContainer instead of overwriting page events (keep it simple)
• [ ] A way to switch/change To languages w/o side panel
• [ ] Inherit font-size from the paragraph and set margin: 0; between paragraphs, use padding-left of first word becomes → text-indent on paragraph

[mekarpeles]

What steps remain on this feature?

1. public release
2. if book language different than browser language, show banner w/ one-click activate translate
3. 40 languages?...