niedev
Custom models
Hi, your app is verrrrrry good. Unfortunately, one feature is missing: custom models.
It would be amazing if the user would be able to import his own custom models (whisper and others). Maybe even set specific ones for specific languages.
Also, when I install the app on my family phones, it is far down in performance compare to mine. Maybe custom models would solve that. Buy honestly I feel like the app needs to have a bunch of performance mods, with custom models.
This comes from a tech guy, not a dev, so I would to hear what do you think.
Hi, thank you!
I have not implemented the possibility to use custom models because the inference of different models happens in different ways, many things are standardized but not all, for example:
- For tokenization (the process of translating input words into numbers) each model not only has a different dictionary but also translates some special symbols differently than the dictionary.
- Regarding the kv-cache (which is not mandatory but increases performance a lot for a model) you need to know some precise values of the model (unique for each model).
- The decoder of each model must always have as first values some particular characters (used to specify some options, for example, the input and output language for the translator, the input language (or languages) for the speech recognition, etc.), and they vary a lot from model to model (not only the initial characters but also their number).
- In walkie talkie mode the app recognizes the audio in both languages at the same time, and each model has different configurations to do in order to do it.
I could have the user specify some of this information in the settings, but they are hard to find (I often had to read source code to find them), also it would be quite complex to implement, and it would not solve the Walkie Talkie mode issue.
The last reason is that I optimized the models themselves by separating some parts and eliminating others, and these optimizations would not be possible for custom models.
However, in the future, I will definitely implement the option to choose other models optimized by me (but not custom ones).
I now tried it again on my family member phone, just now with version 2.0.0. While last time on 2.1.1, it was super slow and did not work. Now it's working on that version with just a little hiccup. Weird
Yeah, the 2 versions use the same models. It's probably because the old GUI is a little lighter, and since that phone CPU is at the limit due to running the models, even that slight change makes a lot of difference. What is the phone model?
Galaxy a51 or a52 i think It's probably the a52... The device is great for the family member use case, unfortunately the performance is really bad. I would really like if you would add a selection in the start of the app, having the option to select between low power mode to the normal one. maybe its better than actually having custom selection of the models?
The thing is, making it choose between two GUIs is very complicated and time consuming to program, and I repeat that in general it wouldn't make that much of a difference for other phones. In fact, the A52 is probably at the limit in terms of the power needed to run RTranslator and so even just a lighter GUI makes a difference, but most phones aren't in that very narrow margin of power (most either can't run RTranslator at all or can do it without any problems). It's like someone who can lift 100kg but can't lift 101, it can happen, but only if your max is 100kg, for everyone else 100 or 101 makes no difference (for better or for worse). Having to choose between different translation/voice recognition models instead is like going from 100kg to 50kg or 200kg, it makes a lot more difference overall.