damus
damus copied to clipboard
damus-io
Automatic offline machine translation
user story
As a Damus user who would like to read notes in foreign languages, I would like my local device to translate notes for me automatically, so that I can seamlessly enjoy reaading notes in foreign languages on my device translated to my preferred language.
acceptance criteria
- there is a method to enable offline local device auto translation (i.e. no need to tap "translate note")
- there is a way to disable
- User is warned if there is a need to download offline language packs
- data size download MB/GB is displayed
- If not on wifi warn user about mobile network bandwidth
Machine translation of notes could be done on-device or otherwise using the language API provided by Apple (which can do offline translations if you download additional data in system settings). ~~Unfortunately Apple does not seem to allow access to their translation API?~~ (Edit: I have seen some activitypub app do this with a button that calls the system translation UI on the note contents, so it seems that is possible at least.)
A nice alternative I found is Google's ML Kit. It is used in another nostr client nostrmo. The translation quality is good and the downloaded models are not very large -- on nostrmo I have loaded 6 languages for translation to English and the app's total documents & data size is still only ~300 MB.
I would love to have offline translations and have spent a significant amount of time investigating. However,
- Apple has not opened up their translation APIs.
- Google's ML Kit for iOS is poorly maintained and does not work for modern development environments. They require Pods but the industry standard is to use Swift Package Manager. It also doesn't work natively for the new Apple M* processors.
I hope things will develop over the next year or so, but nothing is ready at the moment.
On Thu, Jan 18, 2024 at 04:33:00AM -0800, Terry Yiu wrote:
It also doesn't work natively for the new Apple M* processors.
surprised to hear this! that's kind of a deal breaker isn't it.
some activitypub app do this with a button that calls the system translation UI on the note contents
I have no idea how this was implemented, but it would be a great alternative.
I will try to find the app again if it helps.
@alexgleason any idea how Radiant mastodon client does iOS translations?
Whoa, that's definitely Apple native translations. I wonder if they're somehow hacking together some UIKit functionality to get that to work. I'll dig around a bit.
@tyiu just found this on iOS. Have you explored?
I've seen that but I don't think that has anything to do with what Radiant does to expose the Translate sheet.
@bryanmontz do you know how Radiant might be accessing Apple's native translate sheet in-app? Is there some UIKit or accessibility magic going on? I don't think there's an API for it at the moment.
No, I'm not sure how they're triggering Apple's translate sheet, if that's what's happening. I googled around for a minute and didn't see a way to do it.
I wonder if they're somehow hacking together some UIKit functionality to get that to work.
@tyiu I think you are on the right track here. Notice in my screenshot those two big dots at the beginning of the text -- those only appear after translation, and they sort of look like a collapsed form of the text selection cursors. Suggests they are doing some trickery to select the text and then trigger the native translation somehow.
Jake is the dev name. Maybe you could ask him @tyiu
Seems like @jknlsn might be the developer. I've privately messaged him on Mastodon to see if he'd be willing to offer insights.
I wonder if they're somehow hacking together some UIKit functionality to get that to work.
@tyiu I think you are on the right track here. Notice in my screenshot those two big dots at the beginning of the text -- those only appear after translation, and they sort of look like a collapsed form of the text selection cursors. Suggests they are doing some trickery to select the text and then trigger the native translation somehow.
Interesting approach!
Here is the manual flow
Screenshots:
Here is the suggested auto-translate workflow, given you can simulate the manual actions @tyiu
Changing name to automatic offline machine translation
@tyiu solved the underlying technical challenge with new iOS translate API
https://github.com/damus-io/damus/issues/2302
added user story to OP's top entry on this issue. looks like iOS 18 is required for this.
This will be a very interesting feature. iOS18.1 should enable this feature.
