firefox-translations
firefox-translations copied to clipboard
mozilla
https://storage.googleapis.com/bergamot-models-sandbox/ blocked for iranian ips
Describe the bug https://storage.googleapis.com/bergamot-models-sandbox/ is blocked for iranian ips.
and because proxies are slow and unfortunately downloads of models occurs in parallel using proxies to download the models gives timesouts for all of them.
Max time (60000ms) reached while downloading https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/enru/model.enru.intgemm.alphas.bin [backgroundScript.js:722:23](moz-extension://31a56752-e075-4227-a95f-dc18888ee7b9/controller/backgroundScript.js)
12:35:21.336 Max time (60000ms) reached while downloading https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/ruen/model.ruen.intgemm.alphas.bin [backgroundScript.js:722:23](moz-extension://31a56752-e075-4227-a95f-dc18888ee7b9/controller/backgroundScript.js)
12:35:25.597 Max time (60000ms) reached while downloading https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/enru/model.enru.intgemm.alphas.bin [backgroundScript.js:722:23](moz-extension://31a56752-e075-4227-a95f-dc18888ee7b9/controller/backgroundScript.js)
12:35:30.688 Max time (60000ms) reached while downloading https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/enru/model.enru.intgemm.alphas.bin
can the models be downloaded from another place? ipfs is a nice place for static files like these and it has integrity checking too.
or maybe timeouts increased or removed?
Desktop (please complete the following information as possible):
- Firefox Browser version:103.0.1 (64-bit)
- Processor information:x64(64-bit)
- OS version:linux 64
- System type 64-bit
- Firefox Translation Extension version: 1.1.4buildid20220803.111909
The models on storage.googleapis.com are mirrored from github.com/mozilla/firefox-translations-models. You can get them there as well.
I'm not sure about ipfs. All my experiences with it have been slow or unreliable (which is pretty much the opposite as what the idea behind it is…?). The add-on already does integrity checks as well on the downloaded data. Hashes are stored in the model registry that's embedded in the add-on, and are checked every time a model file is downloaded or fetched from the browser cache.
The models on storage.googleapis.com are mirrored from github.com/mozilla/firefox-translations-models. You can get them there as well.
where would I put them?
I'm not sure about ipfs. All my experiences with it have been slow or unreliable (which is pretty much the opposite as what the idea behind it is…?). The add-on already does integrity checks as well on the downloaded data. Hashes are stored in the model registry that's embedded in the add-on, and are checked every time a model file is downloaded or fetched from the browser cache.
that was just a suggestion. I love the idea behind ipfs ( content-addressed instead of location-addressed data) so I suggest it when I can for static data.
can the download be at least be configured to be dont sequentially instead of parallel.
I'm not sure about ipfs. All my experiences with it have been slow or unreliable (which is pretty much the opposite as what the idea behind it is…?). The add-on already does integrity checks as well on the downloaded data. Hashes are stored in the model registry that's embedded in the add-on, and are checked every time a model file is downloaded or fetched from the browser cache.
it is getting better and also it can be a federated node storage. https://www.phoronix.com/news/IPFS-FFmpeg-Open-Source-More
The models on storage.googleapis.com are mirrored from github.com/mozilla/firefox-translations-models. You can get them there as well.
where would I put them?
Apparently browser cache, but I don't think it lines up well. Perhaps the developers claiming its offline could figure out a way to look not only in the browser cache, but a local directory where the models are... That would help me, I've been struggling to up load the models correctly as well...
If there is a better explanation, I'd like to hear it...
The present repository hard codes where models are located.
You may find these open-source models useful: https://github.com/jelmervdl/firefox-translations/releases/ https://translatelocally.com/
The desktop application in particular has support for adding another repository or just having models locally.
Note however that the fa model is poor quality so we have not released it in any software I control. The same applies to a few other low-quality models.
The present repository hard codes where models are located.
You may find these open-source models useful: https://github.com/jelmervdl/firefox-translations/releases/ https://translatelocally.com/
The desktop application in particular has support for adding another repository or just having models locally.
Note however that the fa model is poor quality so we have not released it in any software I control. The same applies to a few other low-quality models.
I dont need the persian model. just any model. for example ru-en.
also why it is hard coded? maybe allow an option for user to use their own models? this also solved my issue too cause I can get my models and load them in addon. this make this addon more versatile and lets users test more models. also this option would help so that this addon wouldn't need to release a version for newer models.
I agree there should be the functionality to add your own models. It's something we're working on, so let me show you the features under development.
Install this instead https://github.com/jelmervdl/firefox-translations/releases/download/v0.6.1/bergamot_translations.xpi . Can you let me know if it works out of the box?
Regarding adding your own model:
- Install the desktop application https://translatelocally.com/ .
- Follow the directions on https://github.com/XapaJIaMnu/translateLocally#importing-custom-models
- Configure the https://github.com/jelmervdl/firefox-translations/releases/download/v0.6.1/bergamot_translations.xpi extension by going to
about:addons, selecting the "Bergamot Translations" extension then click preferences and select TranslateLocally. - Now when you use the extension, translations will be about 16x faster and also allow for any models imported into the desktop application.
I recognize that's not a very usable procedure and we're working on making it smoother.
very great answer. I will try that. last time I tried translatelocally had a compile error on arch from archlinux switching from qt5 to qt6 for compiling.
if I may suggest: use a default that uses the current model, and a custom setting to load models from local files (using custom web urls is easier but would not solve issues like the ones I got with timeout)
tried translatelocally and it run, but it has a bug that it creates infinite popup boxes for asking to download the models from internet that crashes the plasmashell for kde.
@rezad1393 could you open this issue on https://github.com/XapaJIaMnu/translateLocally/issues
Could you also do an ldd translateLocally?
Also can you access https://translatelocally.com/models.json from Iran? If not could you paste the output of curl https://translatelocally.com/models.json. Same for any of the models, eg, https://data.statmt.org/bergamot/models/deen/deen.student.tiny11.tar.gz
@rezad1393 could you open this issue on https://github.com/XapaJIaMnu/translateLocally/issues Could you also do an
ldd translateLocally?Also can you access https://translatelocally.com/models.json from Iran? If not could you paste the output of
curl https://translatelocally.com/models.json. Same for any of the models, eg, https://data.statmt.org/bergamot/models/deen/deen.student.tiny11.tar.gz
I can access both fine.
linux-vdso.so.1 => linux-vdso.so.1 (0x00007ffd55ff1000)
libQt6Widgets.so.6 => /usr/lib/libQt6Widgets.so.6 (0x00007f83a3000000)
libQt6Network.so.6 => /usr/lib/libQt6Network.so.6 (0x00007f83a2e60000)
libQt6Svg.so.6 => /usr/lib/libQt6Svg.so.6 (0x00007f83a5475000)
libarchive.so.13 => /usr/lib/libarchive.so.13 (0x00007f83a53ae000)
libQt6Gui.so.6 => /usr/lib/libQt6Gui.so.6 (0x00007f83a2600000)
libQt6Core.so.6 => /usr/lib/libQt6Core.so.6 (0x00007f83a2000000)
libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007f83a3763000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f83a1c00000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f83a1f18000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f83a538c000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f83a1a19000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f83a54f4000)
libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007f83a1e6f000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00007f83a2e0b000)
libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f83a537c000)
libz.so.1 => /usr/lib/libz.so.1 (0x00007f83a3749000)
libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x00007f83a1600000)
libacl.so.1 => /usr/lib/libacl.so.1 (0x00007f83a5372000)
libexpat.so.1 => /usr/lib/libexpat.so.1 (0x00007f83a2dda000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007f83a25d6000)
liblz4.so.1 => /usr/lib/liblz4.so.1 (0x00007f83a3726000)
libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f83a25c3000)
libEGL.so.1 => /usr/lib/libEGL.so.1 (0x00007f83a25b1000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f83a19c9000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x00007f83a14bd000)
libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f83a1380000)
libQt6DBus.so.6 => /usr/lib/libQt6DBus.so.6 (0x00007f83a190b000)
libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0x00007f83a1339000)
libGLX.so.0 => /usr/lib/libGLX.so.0 (0x00007f83a257f000)
libOpenGL.so.0 => /usr/lib/libOpenGL.so.0 (0x00007f83a1e44000)
libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f83a1302000)
libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f83a1206000)
libmd4c.so.0 => /usr/lib/libmd4c.so.0 (0x00007f83a18f5000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f83a1137000)
libicui18n.so.71 => /usr/lib/libicui18n.so.71 (0x00007f83a0e00000)
libicuuc.so.71 => /usr/lib/libicuuc.so.71 (0x00007f83a0c01000)
libsystemd.so.0 => /usr/lib/libsystemd.so.0 (0x00007f83a0b24000)
libdouble-conversion.so.3 => /usr/lib/libdouble-conversion.so.3 (0x00007f83a0b0e000)
libb2.so.1 => /usr/lib/libb2.so.1 (0x00007f83a0af0000)
libpcre2-16.so.0 => /usr/lib/libpcre2-16.so.0 (0x00007f83a0a5f000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00007f83a0986000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00007f83a0956000)
libcom_err.so.2 => /usr/lib/libcom_err.so.2 (0x00007f83a2575000)
libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00007f83a18e6000)
libkeyutils.so.1 => /usr/lib/libkeyutils.so.1 (0x00007f83a256e000)
libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f83a0944000)
libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f83a0921000)
libGLdispatch.so.0 => /usr/lib/libGLdispatch.so.0 (0x00007f83a086a000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f83a083f000)
libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f83a07c8000)
libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0x00007f83a0775000)
libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f83a0753000)
libicudata.so.71 => /usr/lib/libicudata.so.71 (0x00007f839ea00000)
libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f83a0747000)
libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007f839e8b7000)
libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f839e86e000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f83a1e3d000)
libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f83a073f000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f83a18e1000)
libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f83a0719000)
I assume you're building and using archlinux? The only difference between my build and your build is that mine also links to /usr/lib/libQt6PrintSupport.so.6, which is part to qt6-base. Not sure why it didn't link this for you but also it shouldn't impede any of the functionality.
One way to avoid the first run dialog is to manually download one model and place it inside the same directory as the executable.
Eg:
git clone https://github.com/XapaJIaMnu/translateLocally.git
cd translateLocally
mkdir build
cd build
CC=clang CXX=clang++ cmake ..
make -j8
wget https://data.statmt.org/bergamot/models/deen/deen.student.tiny11.tar.gz
tar -xvf deen.student.tiny11.tar.gz
./translateLocally
Alternatively, you can manually create a configuration directory to avoid the first run dialog:
mkdir -p ~/.config/translateLocally
cd ~/.config/translateLocally
wget https://data.statmt.org/bergamot/models/deen/deen.student.tiny11.tar.gz
tar -xvf deen.student.tiny11.tar.gz
And then run translateLocally.
To be clear, the issue is when you run translateLocally standalone, and not when you try to launch it from the browser?
yes , "I use arch btw"⟨™⟩ thank you for your help. you are the owner of translateloccaly git right? cause this app wouldnt compile on my machien because archlinux seems to change from qt5 to qt6 for default compile. the aur https://aur.archlinux.org/packages/translatelocally-git owner said he fixed it. now it compiles but has this bug.
I used your fix and it seem to work now. btw you missed "cd ~/.config/translateLocally" I think.
thank you. now I can test this more.
@rezad1393 I'm Dheart from arch forums, you are talking to the same person.
I use arch, btw.
@rezad1393 I'm Dheart from arch forums, you are talking to the same person.
I use arch, btw.
I yearn for the day that our beloved lawmakers make it illegal to have aliases and every one would have only "one true identity" so that stuff like this would not happen. /s
@rezad1393 could you open this issue on https://github.com/XapaJIaMnu/translateLocally/issues Could you also do an
ldd translateLocally?Also can you access https://translatelocally.com/models.json from Iran? If not could you paste the output of
curl https://translatelocally.com/models.json. Same for any of the models, eg, https://data.statmt.org/bergamot/models/deen/deen.student.tiny11.tar.gz
btw where are the addon repository for language models? because the one in the app doesn't have Russian. is there a url I can add for repo like this https://translatelocally.com/models.json or is that just for that app and I have to download them and import them locally? if the later then where are the russian or persian models?
Hi,
We support external repos. You can add an extra repository through the settings menu and one such repository is OpusMT: https://object.pouta.csc.fi/OPUS-MT-models/app/models.json
In terms of Farsi, we have Farsi-English, but we haven't officially released it. You can find it here and you can package it and import it in translateLocally: https://github.com/browsermt/students/tree/master/faen
You can get additional models from mozilla's repository: https://github.com/mozilla/firefox-translations-models They do not have the necessary configuration files, but you can easily build one yourself, following examples from already downloaded models and these instructions: https://github.com/XapaJIaMnu/translateLocally#importing-custom-models
It's annoying, we're sorry ; (. We will work making it better in the future.
Thanks. I will try with translate locally although there are some. Restrictions in place that will limit my ability to install. I have tried already importing the bin files to the Firefox browser cache and it won't collect it. I have tried to debug the extension to look for the cache path and fcuked up getting the right directories. It seems the claim that its offline is not true until you can bundle a model with the extension, or add a parameter in the extension to select a folder where it lives. Thanks I look forward to this support in the future
