pontoon
pontoon copied to clipboard
mozilla
Refactor sync
This is effectively a rewrite of the sync_project() function that's currently here, and which ends up calling most of the code under pontoon/sync/.
The end results of the code here should be the same as currently, but the implementation is completely new, and does things in a different order.
Per-locale repositories are dropped here, as per #3303.
Explicitly left out of this PR but liable to change later:
- What sync logging information is persisted in the database.
- How files are read and written.
- How localizable messages are represented in the database.
- How aggregated stats are gathered.
Currently, Pretranslation is tightly integrated with sync in order to minimize time between exposing new strings for localization and pretranslating them. What's your plan with this?
Ah, I'd missed that! Yeah, that needs to happen the same as before.
Codecov Report
Attention: Patch coverage is 90.05266% with 170 lines in your changes missing coverage. Please review.
Project coverage is 79.20%. Comparing base (
e3e7168) to head (ff2b00d). Report is 9 commits behind head on main.
Additional details and impacted files
This is now at the dangerous stage of looking like it works. But some verification work still remains:
- [x] Add tests for the fixtures used by the old sync tests.
- [x] Check the sync results against the active pontoon.mozilla.org projects.
I've now manually validated the refactor by running it against the last three months or so of changes on pontoon.mozilla.org and fixing all of the issues that initially arose, until all of it passed in one go. That last run was about 3.5 hours, of which about 2h was spent on common-voice, 1h on firefox, and half an hour for all the rest. Some parts are definitely optimizable into better shape, in particular the aggregated-stats update, but I think the performance is now sufficient for a first pass.
I've looked at about half of the changes and this looks like a massive improvement over the current code! 🥳
I haven't tested sync yet, but I'm having some issues downloading files on stage and locally.
Stage / AMO Linter:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/site-packages/newrelic/hooks/framework_django.py", line 529, in wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/base/views.py", line 768, in download_translations
content, filename = download_translations_zip(project, locale)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/sync/utils.py", line 29, in download_translations_zip
lc_path = paths.format_target_path(tgt_path, locale.code)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/site-packages/moz/l10n/paths/discover.py", line 237, in format_target_path
return normpath(join(base, target.format(locale=dir)))
^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'format'
10.1.55.79 - - [15/Oct/2024:11:28:52 +0000] "GET /translations/?code=sl&slug=amo-linter&part=LC_MESSAGES/messages.po HTTP/1.1" 500 4290 "https://mozilla-pontoon-staging.herokuapp.com/sl/amo-linter/LC_MESSAGES/messages.po/?string=175132" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
Exception ignored in: <function ZipFile.__del__ at 0x7fbb5a373060>
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.11/zipfile.py", line 1895, in __del__
self.close()
File "/app/.heroku/python/lib/python3.11/zipfile.py", line 1912, in close
self.fp.seek(self.start_dir)
ValueError: I/O operation on closed file.
Oct 15 13:28:53 mozilla-pontoon-staging heroku/router at=info method=GET path="/ann/terminology/project-info?&label=%27%2fxid=1%20&highlight=%27%2fxid=1%20&yz=%27%2fxid=1%20&feature=%27%2fxid=1%20&redlink=%27%2fxid=1%20&args=%27%2fxid=1%20&before=%27%2fxid=1%20&accountID=%27%2fxid=1%20&site=%27%2fxid=1%20&tc=%27%2fxid=1%20&gs=%27%2fxid=1%20&itemprop=%27%2fxid=1%20&matchtype=%27%2fxid=1%20&dj=%27%2fxid=1%20&toast=%27%2fxid=1%20&token=%27%2fxid=1%20&pais=%27%2fxid=1%20&qo=%27%2fxid=1%20&cat=%27%2fxid=1%20&ja=%27%2fxid=1%20&od=%27%2fxid=1%20&fb_action_ids=%27%2fxid=1%20&qt=%27%2fxid=1%20&pq=%27%2fxid=1%20&photoId=%27%2fxid=1%20&releaseid=%27%2fxid=1%20&filtertype_9=%27%2fxid=1%20&nurlen=%27%2fxid=1%20&jsessionid=%27%2fxid=1%20&text=%27%2fxid=1%20&origin_url=%27%2fxid=1%20&PUT=%27%2fxid=1%20&wd=%27%2fxid=1%20&fromUrl=%27%2fxid=1%20&like_comment=%27%2fxid=1%20&redirectTo=%27%2fxid=1%20&traffic_src=%27%2fxid=1%20&header=%27%2fxid=1%20&update=%27%2fxid=1%20&targetid=%27%2fxid=1%20" host=mozilla-pontoon-staging.herokuapp.com request_id=85e36ffb-48ae-4eb7-85d6-1832010697a1 fwd="185.187.235.142" dyno=web.1 connect=0ms service=46ms status=429 bytes=2152 protocol=https
Stage / Firefox:
Internal Server Error: /translations/
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/site-packages/newrelic/hooks/framework_django.py", line 529, in wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/base/views.py", line 768, in download_translations
content, filename = download_translations_zip(project, locale)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/sync/utils.py", line 19, in download_translations_zip
checkouts = checkout_repos(project)
^^^^^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/sync/core/checkout.py", line 95, in checkout_repos
target = Checkout(project.slug, repo, pull, force)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/pontoon/sync/core/checkout.py", line 39, in __init__
versioncontrol.update(self.url, self.path, db_repo.branch)
File "/app/pontoon/sync/repositories/hg.py", line 23, in update
raise PullFromRepositoryException(error)
pontoon.sync.repositories.utils.PullFromRepositoryException: No such file or directory
Oct 15 13:26:51 mozilla-pontoon-staging heroku/router at=info method=GET path="/translations/?code=sl&slug=firefox&part=mobile/android/chrome/browser.properties" host=mozilla-pontoon-staging.herokuapp.com request_id=5c1e80e7-9d10-4f4e-a1ac-da9d6b652f5d fwd="31.15.231.4" dyno=web.1 connect=1ms service=774ms status=500 bytes=5233 protocol=https
Localhost / AMO Linter:
[server] [DEBUG:pontoon.sync.repositories.git] 2024-10-15 11:29:04,158 Git: Updating repo [email protected]:mozilla/addons-linter.git
[server] [DEBUG:pontoon.sync.repositories.git] 2024-10-15 11:29:04,159 Git: Cloning repo...
[server] [DEBUG:pontoon.sync.repositories.git] 2024-10-15 11:29:04,172 b''
[server] [ERROR:django.request] 2024-10-15 11:29:04,629 Internal Server Error: /translations/
[server] Traceback (most recent call last):
[server] File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
[server] response = get_response(request)
[server] ^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
[server] response = wrapped_callback(request, *callback_args, **callback_kwargs)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
[server] return func(*args, **kwds)
[server] ^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/base/views.py", line 768, in download_translations
[server] content, filename = download_translations_zip(project, locale)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/utils.py", line 19, in download_translations_zip
[server] checkouts = checkout_repos(project)
[server] ^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/core/checkout.py", line 95, in checkout_repos
[server] target = Checkout(project.slug, repo, pull, force)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/core/checkout.py", line 39, in __init__
[server] versioncontrol.update(self.url, self.path, db_repo.branch)
[server] File "/app/pontoon/sync/repositories/git.py", line 33, in update
[server] raise PullFromRepositoryException(error)
[server] pontoon.sync.repositories.utils.PullFromRepositoryException: Cloning into '/app/media/projects/amo-linter/[email protected]:mozilla/addons-linter.git'...
[server] /home/pontoon/.ssh/config: line 6: Bad configuration option: usekeychain
[server] /home/pontoon/.ssh/config: terminating, 1 bad configuration options
[server] fatal: Could not read from remote repository.
[server]
[server] Please make sure you have the correct access rights
[server] and the repository exists.
[server]
[server] [ERROR:django.server] 2024-10-15 11:29:04,632 "GET /translations/?code=sl&slug=amo-linter&part=LC_MESSAGES/messages.po HTTP/1.1" 500 18449
Localhost / AMO Linter:
That seems like a bad configuration on my end. Relatedly, it would be nice to add a section to developer docs on how to configure SSH keys so that sync works (once needs a passwordless key set up in ~/.ssh/config).
Now the error is the same as on stage for AMO Linter.
I can now download files from AMO Linter, but downloading from Firefox still fails on stage (it works locally).
Locally I couldn't make upload work, though. I downloaded the brand.ftl file, deleted a string in Pontoon, uploaded the file, and the deleted string didn't get imported.
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737405&selected=1782867344000737405) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737405&selected=1782867344000737405) Traceback (most recent call last):
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737406&selected=1782867344000737406) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737406&selected=1782867344000737406) File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737408&selected=1782867344000737408) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737408&selected=1782867344000737408) response = get_response(request)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737410&selected=1782867344000737410) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737410&selected=1782867344000737410) ^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737412&selected=1782867344000737412) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737412&selected=1782867344000737412) File "/app/.heroku/python/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737414&selected=1782867344000737414) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737414&selected=1782867344000737414) response = wrapped_callback(request, *callback_args, **callback_kwargs)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737415&selected=1782867344000737415) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737415&selected=1782867344000737415) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737417&selected=1782867344000737417) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737417&selected=1782867344000737417) File "/app/.heroku/python/lib/python3.11/site-packages/newrelic/hooks/framework_django.py", line 529, in wrapper
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737418&selected=1782867344000737418) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737418&selected=1782867344000737418) return wrapped(*args, **kwargs)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737420&selected=1782867344000737420) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737420&selected=1782867344000737420) ^^^^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737421&selected=1782867344000737421) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737421&selected=1782867344000737421) File "/app/.heroku/python/lib/python3.11/contextlib.py", line 81, in inner
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737423&selected=1782867344000737423) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737423&selected=1782867344000737423) return func(*args, **kwds)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737424&selected=1782867344000737424) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737424&selected=1782867344000737424) ^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737426&selected=1782867344000737426) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737426&selected=1782867344000737426) File "/app/pontoon/base/views.py", line 768, in download_translations
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737427&selected=1782867344000737427) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737427&selected=1782867344000737427) content, filename = download_translations_zip(project, locale)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737429&selected=1782867344000737429) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737429&selected=1782867344000737429) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737430&selected=1782867344000737430) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737430&selected=1782867344000737430) File "/app/pontoon/sync/utils.py", line 19, in download_translations_zip
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737432&selected=1782867344000737432) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737432&selected=1782867344000737432) checkouts = checkout_repos(project)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737433&selected=1782867344000737433) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737433&selected=1782867344000737433) ^^^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737435&selected=1782867344000737435) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737435&selected=1782867344000737435) File "/app/pontoon/sync/core/checkout.py", line 95, in checkout_repos
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737436&selected=1782867344000737436) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737436&selected=1782867344000737436) target = Checkout(project.slug, repo, pull, force)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737438&selected=1782867344000737438) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737438&selected=1782867344000737438) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737439&selected=1782867344000737439) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737439&selected=1782867344000737439) File "/app/pontoon/sync/core/checkout.py", line 39, in __init__
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737442&selected=1782867344000737442) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737442&selected=1782867344000737442) versioncontrol.update(self.url, self.path, db_repo.branch)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737443&selected=1782867344000737443) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737443&selected=1782867344000737443) File "/app/pontoon/sync/repositories/hg.py", line 23, in update
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737445&selected=1782867344000737445) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737445&selected=1782867344000737445) raise PullFromRepositoryException(error)
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737446&selected=1782867344000737446) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737446&selected=1782867344000737446) pontoon.sync.repositories.utils.PullFromRepositoryException: No such file or directory
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737451&selected=1782867344000737451) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737451&selected=1782867344000737451) Too Many Requests: /en-GB//dialin.teams
Oct 15 17:25:21 [mozilla-pontoon-staging](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=host%3A%22mozilla-pontoon-staging%22&focus=1782867344000737453&selected=1782867344000737453) [app/web.1](https://my.na-01.cloud.solarwinds.com/218426335867993088/logs?q=program%3A%22app%2Fweb.1%22&focus=1782867344000737453&selected=1782867344000737453) 10.1.94.198 - - [15/Oct/2024:15:25:21 +0000] "GET /translations/?code=sl&slug=firefox&part=browser/branding/official/brand.ftl HTTP/1.1" 500 4212 "https://mozilla-pontoon-staging.herokuapp.com/sl/firefox/browser/branding/official/brand.ftl/?string=192265" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
I've now experienced first slow sync. I'm on a 2 days old DB dump. A few new strings were added to AMO today and I synced AMO for the first time locally. The following operations are slow:
Compiling updates takes 5 minutes:
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:11,362 [amo] Compiling updates...
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:45,428 [amo] Syncing translations from repo...
Updating stats takes 11 minutes (running calculate_stats() on all TRs and aggregate_stats() on all Project, Locale and ProjectLocale objects take a few seconds):
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:52,108 [amo] Updating stats for 3039 changes
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:17:43,297 update_stats: 70 translated resources
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:22:21,550 update_stats: 42 projectlocales
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:24:06,456 update_stats: 1 project
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:26:54,425 update_stats: 42 locales
[INFO:pontoon.sync.core] 2024-10-15 14:26:54,966 [amo] Notifying 1323 users about 4 new strings
Updating all locales takes a minute:
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:26:58,534 [amo:LC_MESSAGES/django.po] Updating all locales
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:27:47,279 [amo] Skipping commit & push
Full log:
pontoon@49efeb27098c:/app$ ./manage.py sync_projects --projects=amo --no-commit
Scheduling sync for project AMO.
[INFO:pontoon.sync.core] 2024-10-15 14:07:02,518 [amo] Sync start
[DEBUG:pontoon.sync.repositories.git] 2024-10-15 14:07:02,522 Git: Updating repo [email protected]:mozilla/olympia.git
[DEBUG:pontoon.sync.repositories.git] 2024-10-15 14:07:02,523 Git: Cloning repo...
[DEBUG:pontoon.sync.repositories.git] 2024-10-15 14:09:43,747 Git: Repo cloned.
[INFO:pontoon.sync.core.checkout] 2024-10-15 14:09:43,751 [amo] Repo updated from cd974ab87808b380a8c16f34cf6b630516fe3a8b to e207ca26fd
[DEBUG:pontoon.sync.core.checkout] 2024-10-15 14:09:43,754 [amo] target root: /app/media/projects/amo/[email protected]:mozilla/olympia.git
[DEBUG:pontoon.sync.core.paths] 2024-10-15 14:09:43,771 [amo] Paths(auto): ref_root=locale/templates base=locale
[INFO:pontoon.sync.core.entities] 2024-10-15 14:09:43,787 [amo] Syncing entities from repo...
[INFO:pontoon.sync.core.entities] 2024-10-15 14:09:56,297 [amo] Source entity updates: added 4
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:56,323 [amo] Reading changes from 191 changed target files
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:56,323 [amo] Scanning for translation updates...
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:58,015 [amo:../package-lock.json] Not an L10n target path
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:58,015 [amo:../package.json] Not an L10n target path
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:58,017 [amo] Filtering matches from translations...
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:59,258 [amo] Filtering matches from translations... 1/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:09:59,641 [amo] Filtering matches from translations... 2/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:00,233 [amo] Filtering matches from translations... 3/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:00,816 [amo] Filtering matches from translations... 4/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:01,386 [amo] Filtering matches from translations... 5/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:01,911 [amo] Filtering matches from translations... 6/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:02,473 [amo] Filtering matches from translations... 7/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:03,010 [amo] Filtering matches from translations... 8/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:03,533 [amo] Filtering matches from translations... 9/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:04,068 [amo] Filtering matches from translations... 10/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:04,628 [amo] Filtering matches from translations... 11/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:05,157 [amo] Filtering matches from translations... 12/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:05,716 [amo] Filtering matches from translations... 13/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:06,263 [amo] Filtering matches from translations... 14/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:06,827 [amo] Filtering matches from translations... 15/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:07,444 [amo] Filtering matches from translations... 16/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:07,993 [amo] Filtering matches from translations... 17/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:08,546 [amo] Filtering matches from translations... 18/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:09,119 [amo] Filtering matches from translations... 19/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:09,670 [amo] Filtering matches from translations... 20/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:10,232 [amo] Filtering matches from translations... 21/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:10,800 [amo] Filtering matches from translations... 22/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:11,355 [amo] Filtering matches from translations... 23/23
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:11,355 [amo] Filtering db changes from translations...
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:10:11,362 [amo] Compiling updates...
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:45,428 [amo] Syncing translations from repo...
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:50,861 [amo] Rejected 3602 translations from repo changes
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:51,712 [amo] Approved 6 translations from repo changes
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:51,723 [amo] Created 35 translations from repo changes
[INFO:pontoon.sync.core.translations_from_repo] 2024-10-15 14:15:52,108 [amo] Updating stats for 3039 changes
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:17:43,297 update_stats: 70 translated resources
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:22:21,550 update_stats: 42 projectlocales
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:24:06,456 update_stats: 1 project
[DEBUG:pontoon.base.models.translated_resource] 2024-10-15 14:26:54,425 update_stats: 42 locales
[INFO:pontoon.sync.core] 2024-10-15 14:26:54,966 [amo] Notifying 1323 users about 4 new strings
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:26:56,028 [amo] Updating 2 changed resources
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:26:56,029 [amo:LC_MESSAGES/djangojs.po] Updating all locales
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:26:58,534 [amo:LC_MESSAGES/django.po] Updating all locales
[INFO:pontoon.sync.core.translations_to_repo] 2024-10-15 14:27:47,279 [amo] Skipping commit & push
[INFO:pontoon.sync.core] 2024-10-15 14:27:48,255 [amo] Sync done
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:48,257 Fetching pretranslations for project AMO started
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:48,959 Fetching pretranslations for locale cs started
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,078 Fetching pretranslations for locale cs done: No pretranslation fetched
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,078 Fetching pretranslations for locale nl started
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,160 Fetching pretranslations for locale nl done: No pretranslation fetched
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,161 Fetching pretranslations for locale pt-BR started
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,174 Fetching pretranslations for locale pt-BR done: No pretranslation fetched
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,175 Fetching pretranslations for locale sl started
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,213 Fetching pretranslations for locale sl done: No pretranslation fetched
[INFO:pontoon.pretranslation.tasks] 2024-10-15 14:27:49,213 Fetching pretranslations for project AMO done
I can now download files from AMO Linter, but downloading from Firefox still fails on stage (it works locally).
From my own experimentation, it looks like there's at least some timing issue due to the git clone that's now needed. Retrying got amo-frontend download to work for me.
Locally I couldn't make upload work, though. I downloaded the brand.ftl file, deleted a string in Pontoon, uploaded the file, and the deleted string didn't get imported.
Does this work even now? As in, does uploading a resource with a string removed remove it from Pontoon?
Does this work even now? As in, does uploading a resource with a string removed remove it from Pontoon?
I uploaded a resource which had a translation that didn't exist in Pontoon.
From my own experimentation, it looks like there's at least some timing issue due to the git clone that's now needed. Retrying got
amo-frontenddownload to work for me.
Can we use git clone --depth 1 for downloading files?
Not sure if this is worth investigating, as it's a messy local setup. I had a local sync from before testing this code, but the remote branch was removed and changed (so, the latest changeset in Pontoon is not valid).
That explains this error
[ERROR:pontoon.sync.repositories.git] 2024-10-16 07:00:25,534 Error while executing command `['git', 'diff', '--name-status', '--find-renames=100%', 'e19173e24d34637c96af7413f4a22ef953bef2db..HEAD', '--', '/app/media/projects/firefox-ssh/mozilla-l10n/firefox-l10n-source.git']` in `/app/media/projects/firefox-ssh/mozilla-l10n/firefox-l10n-source.git`: fatal: Invalid revision range e19173e24d34637c96af7413f4a22ef953bef2db..HEAD
[server]
[WARNING:pontoon.sync.core.checkout] 2024-10-16 07:00:25,534 [firefox-ssh] Considering all files as changed
But then I end up with a traceback about stats
[pg] 2024-10-16 07:06:10.072 UTC [18337] ERROR: new row for relation "base_locale" violates check constraint "base_locale_approved_strings_check"
[pg] 2024-10-16 07:06:10.072 UTC [18337] DETAIL: Failing row contains (50, 6, -4, 0, 0, 0, 0, it, it, it, it-it, , Italian, (n != 1), 1,5, Latin, ltr, 69025000, , 13250, 100, 99, , , , t).
[pg] 2024-10-16 07:06:10.072 UTC [18337] STATEMENT: UPDATE "base_locale" SET "total_strings" = ("base_locale"."total_strings" + 3), "approved_strings" = ("base_locale"."approved_strings" + -4), "pretranslated_strings" = ("base_locale"."pretranslated_strings" + 0), "strings_with_errors" = ("base_locale"."strings_with_errors" + 0), "strings_with_warnings" = ("base_locale"."strings_with_warnings" + 0), "unreviewed_strings" = ("base_locale"."unreviewed_strings" + 0) WHERE "base_locale"."id" = 50
[server] [ERROR:pontoon.base.errors] 2024-10-16 07:06:10,084 new row for relation "base_locale" violates check constraint "base_locale_approved_strings_check"
[server] DETAIL: Failing row contains (50, 6, -4, 0, 0, 0, 0, it, it, it, it-it, , Italian, (n != 1), 1,5, Latin, ltr, 69025000, , 13250, 100, 99, , , , t).
[server] Traceback (most recent call last):
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
[server] return self.cursor.execute(sql, params)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] psycopg2.errors.CheckViolation: new row for relation "base_locale" violates check constraint "base_locale_approved_strings_check"
[server] DETAIL: Failing row contains (50, 6, -4, 0, 0, 0, 0, it, it, it, it-it, , Italian, (n != 1), 1,5, Latin, ltr, 69025000, , 13250, 100, 99, , , , t).
[server]
[server]
[server] The above exception was the direct cause of the following exception:
[server]
[server] Traceback (most recent call last):
[server] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
[server] R = retval = fun(*args, **kwargs)
[server] ^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/tasks.py", line 44, in sync_project_task
[server] sync_project(project, sync_log, pull=pull, commit=commit, force=force)
[server] File "/app/pontoon/sync/core/__init__.py", line 50, in sync_project
[server] added_entities_count, changed_paths, removed_paths = sync_entities_from_repo(
[server] ^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/core/entities.py", line 73, in sync_entities_from_repo
[server] old_res_added_ent_count, changed_paths = update_resources(project, updates, now)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/app/pontoon/sync/core/entities.py", line 213, in update_resources
[server] tr.calculate_stats()
[server] File "/app/pontoon/base/models/translated_resource.py", line 325, in calculate_stats
[server] self.adjust_all_stats(
[server] File "/app/pontoon/base/models/translated_resource.py", line 187, in adjust_all_stats
[server] locale.adjust_stats(*args, **kwargs)
[server] File "/app/pontoon/base/models/aggregated_stats.py", line 83, in adjust_stats
[server] self.save(
[server] File "/app/pontoon/base/models/locale.py", line 446, in save
[server] super().save(*args, **kwargs)
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
[server] self.save_base(
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
[server] updated = self._save_table(
[server] ^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 990, in _save_table
[server] updated = self._do_update(
[server] ^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1054, in _do_update
[server] return filtered._update(values) > 0
[server] ^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1231, in _update
[server] return query.get_compiler(self.db).execute_sql(CURSOR)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1984, in execute_sql
[server] cursor = super().execute_sql(result_type)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
[server] cursor.execute(sql, params)
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 102, in execute
[server] return super().execute(sql, params)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
[server] return self._execute_with_wrappers(
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
[server] return executor(sql, params, many, context)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
[server] with self.db.wrap_database_errors:
[server] File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
[server] raise dj_exc_value.with_traceback(traceback) from exc_value
[server] File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
[server] return self.cursor.execute(sql, params)
[server] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[server] django.db.utils.IntegrityError: new row for relation "base_locale" violates check constraint "base_locale_approved_strings_check"
[server] DETAIL: Failing row contains (50, 6, -4, 0, 0, 0, 0, it, it, it, it-it, , Italian, (n != 1), 1,5, Latin, ltr, 69025000, , 13250, 100, 99, , , , t).
[server]
The stats update should now be much more robust, and a bit simpler -- albeit written in SQL, as I couldn't figure out the corresponding Django invocations. In particular the COUNT(*) FILTER (WHERE ...) values are probably not even possible.
This also guarantees that the work is entirely done within the database, where even for the most complex projects it should take at most a few seconds even though the update is applied to the entire current project.
A "messy local setup" as mentioned above should no longer be able to produce the error that @flodolo encountered.
As a minor regression, gettext plurals are now counted as separate entries. This means that when translating from English to a locale with more plural categories may result in an approved_strings count greater than total_strings. This will get fixed by my subsequent work, which will change gettext plurals are stored in the database.
A "messy local setup" as mentioned above should no longer be able to produce the error that @flodolo encountered.
Confirming that this worked locally 👍🏼
Confirming that this worked locally 👍🏼
Correction: the error now shows up when you update any string (e.g. reject one), e.g.
2024-10-16 15:07:05 server-1 | django.db.utils.IntegrityError: new row for relation "base_locale" violates check constraint "base_locale_approved_strings_check"
2024-10-16 15:07:05 server-1 | DETAIL: Failing row contains (50, 0, -1, 0, 0, 0, 1, it, it, it, it-it, , Italian, (n != 1), 1,5, Latin, ltr, 69025000, , 25173, 100, 99, , , , t).
The stats update should now be much more robust, and a bit simpler -- albeit written in SQL, as I couldn't figure out the corresponding Django invocations. In particular the
COUNT(*) FILTER (WHERE ...)values are probably not even possible.As a minor regression, gettext plurals are now counted as separate entries. This means that when translating from English to a locale with more plural categories may result in an
approved_stringscount greater thantotal_strings. This will get fixed by my subsequent work, which will change gettext plurals are stored in the database.
Note that calculate_stats() is called from many places (including the translate view) and that approved_strings greater than total_strings might cause our charts (in dashboards as well as in translate view) to break (I didn't actually test). All that means we shouldn't just thoroughly test the sync module, but the entire app.
Two issues with pretranslation:
- Local setup with Firefox + Italian, a new file is available on sync (profiles.ftl) but it's not pretranslated.
- Manual pretranslation is broken
2024-10-30 07:43:20 Traceback (most recent call last):
2024-10-30 07:43:20 File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
2024-10-30 07:43:20 response = get_response(request)
2024-10-30 07:43:20 ^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:43:20 File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
2024-10-30 07:43:20 response = wrapped_callback(request, *callback_args, **callback_kwargs)
2024-10-30 07:43:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:43:20 File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
2024-10-30 07:43:20 return view_func(request, *args, **kwargs)
2024-10-30 07:43:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:43:20 File "/app/pontoon/base/utils.py", line 95, in wrap
2024-10-30 07:43:20 return f(request, *args, **kwargs)
2024-10-30 07:43:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:43:20 File "/app/pontoon/administration/views.py", line 561, in manually_pretranslate_project
2024-10-30 07:43:20 pretranslate.delay(project.pk)
2024-10-30 07:43:20 ^^^^^^^^^^^^^^^^^^
2024-10-30 07:43:20 AttributeError: 'function' object has no attribute 'delay'
Also for the same new file (not committed to the repo). Trying to download translations will crash Pontoon, because the file is not in the repo.
2024-10-30 07:56:43 [ERROR:django.request] 2024-10-30 06:56:43,698 Internal Server Error: /translations/
2024-10-30 07:56:43 Traceback (most recent call last):
2024-10-30 07:56:43 File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
2024-10-30 07:56:43 response = get_response(request)
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
2024-10-30 07:56:43 response = wrapped_callback(request, *callback_args, **callback_kwargs)
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner
2024-10-30 07:56:43 return func(*args, **kwds)
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 File "/app/pontoon/base/views.py", line 768, in download_translations
2024-10-30 07:56:43 content, filename = download_translations_zip(project, locale)
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 File "/app/pontoon/sync/utils.py", line 31, in download_translations_zip
2024-10-30 07:56:43 zipfile.write(lc_path, basename(tgt_path))
2024-10-30 07:56:43 File "/usr/local/lib/python3.11/zipfile.py", line 1796, in write
2024-10-30 07:56:43 zinfo = ZipInfo.from_file(filename, arcname,
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 File "/usr/local/lib/python3.11/zipfile.py", line 535, in from_file
2024-10-30 07:56:43 st = os.stat(filename)
2024-10-30 07:56:43 ^^^^^^^^^^^^^^^^^
2024-10-30 07:56:43 FileNotFoundError: [Errno 2] No such file or directory: '/app/media/projects/firefox/[email protected]:flodolo/firefox-l10n.git/it/mobile/android/mobile-l10n.js'
EDIT: it actually fails also after syncing to the repo, so the file is actually there.
~Looks like I'm once again unable to set up Firefox from scratch (sync doesn't pick up translations, even with --force).~
~The log is full of messages like about the TOML config~
[DEBUG:pontoon.sync.core.translations_from_repo] 2024-10-31 11:20:03,375 [firefox:../../[email protected]:mozilla-l10n/firefox-l10n.git/it/devtools/client/toolbox.ftl] Not an L10n target path
Update: Not sure what happened, but I think my local DB is toasted (too many tests). Creating a new project with a different slug worked 🤷🏼
I don't think these should be closed:
- Closes https://github.com/mozilla/pontoon/issues/2023 -- entity- and translation-level changes are logged in ActionLog
- Closes https://github.com/mozilla/pontoon/issues/2110 -- the remaining sync tests are largely replaced here
I have questions about these:
- Fixes https://github.com/mozilla/pontoon/issues/2068
- If Flod translates some string for Italian before en-US files are updated, pushes them to GitHub directly, then Pontoon syncs, then after a while we merge the new source strings to the en-US repo finally... will Pontoon import Flod's translations on next sync?
- Fixes https://github.com/mozilla/pontoon/issues/2087 -- git file moves are caught, but not copies
- Just to double check, this is https://github.com/mozilla/pontoon/issues/2211 plus the ability to keep translation metadata and unreviewed suggestions?
- Fixes https://github.com/mozilla/pontoon/issues/3449
- This breaks in
reset_active_translations(), which is still in. Do we reset active translations on upload differently?
- This breaks in
I don't think these should be closed:
- Closes More detailed sync log #2023 -- entity- and translation-level changes are logged in ActionLog
- Closes Shift existing Pontoon tests to py.test #2110 -- the remaining sync tests are largely replaced here
Dropped the closing of these issues.
I have questions about these:
- Fixes On source repo change, import translations from corresponding l10n files #2068
If Flod translates some string for Italian before en-US files are updated, pushes them to GitHub directly, then Pontoon syncs, then after a while we merge the new source strings to the en-US repo finally... will Pontoon import Flod's translations on next sync?
No; dropped this too from the list of issues being closed. As is, the it repo changes would only get picked up if/when the file changes at a time when the source strings are also available.
- Fixes Detect moved and copied reference files, move or copy l10n files #2087 -- git file moves are caught, but not copies
Just to double check, this is Detect moved/renamed files in git as changed files #2211 plus the ability to keep translation metadata and unreviewed suggestions?
Yes.
- Fixes Server error when uploading translations (messages.json) #3449
This breaks inreset_active_translations(), which is still in. Do we reset active translations on upload differently?
Yes, that's now handled here.
- Fixes On source repo change, import translations from corresponding l10n files #2068 If Flod translates some string for Italian before en-US files are updated, pushes them to GitHub directly, then Pontoon syncs, then after a while we merge the new source strings to the en-US repo finally... will Pontoon import Flod's translations on next sync?
No; dropped this too from the list of issues being closed. As is, the
itrepo changes would only get picked up if/when the file changes at a time when the source strings are also available.
I'm not completely sure why, but I haven't seen the problem since we moved to the monorepo for Firefox.
Can it be that Pontoon sees new strings in a file for en-US, and re-read the file for Italian as soon as there is an update to any other locale?
I'm not completely sure why, but I haven't seen the problem since we moved to the monorepo for Firefox.
Can it be that Pontoon sees new strings in a file for en-US, and re-read the file for Italian as soon as there is an update to any other locale?
I got uncertain about my presumptions, and wrote a test case to validate the behaviour -- see the preceding commit. With the new sync, Pontoon applies changes to target files eagerly, which means that a translation added before it's available in the source seen by Pontoon will be removed from the target file, because it doesn't correspond to a source string.
I hope the comments I added inline are visible.
I hit some issues with file download/upload on stage:
- I've added a Pontoon Test project, and get
FileNotFoundErrorwhen I try to download the file. - The app crashes when I try to download a file from the Firefox project.
- I managed to download AMO file (in 2nd attempt). I made a change to one of the strings and uploaded it. That stats went off.
I'm not completely sure why, but I haven't seen the problem since we moved to the monorepo for Firefox.
Can it be that Pontoon sees new strings in a file for en-US, and re-read the file for Italian as soon as there is an update to any other locale?
Thanks for this insight. I looked more closely and this bug was no longer reproducible in a single-locale scenario. Closed #2068 and added https://github.com/mozilla/pontoon/issues/2068#issuecomment-2527789436.
* The app crashes when I try to download a file from the Firefox project.
Locally for me, downloading from a WebExtension project (VPN) worked, downloading from mozilla.org crashed after what felt like a long pause.
[Errno 2] No such file or directory: '/app/media/projects/mozillaorg/[email protected]:flodolo/www-l10n.git/it/firefox/welcome/page20-21.ftl'
Looks like it's searching for a file that is not available for Italian (because of the config). So, it probably shouldn't look for it in the first place.
Firefox also fails, even without config, because of a missing file
No such file or directory: '/app/media/projects/firefox-local/[email protected]:mozilla-l10n/firefox-l10n.git/it/mobile/android/mobile-l10n.js'
Uploading worked for the VPN extension and mozilla.org (didn't try Firefox).
To add to my last comment, the behavior is clearly changing, and I'm not sure it's the right thing to do: if I'm on the brand.ftl Firefox, in current Pontoon I only get that file. I have the feeling Pontoon is now tring to serve me the whole of Firefox in a ZIP file (thinking back, I was surprised to get a ZIP file from the VPN extension, which is only one file).
Switching to shallow (--depth 1) cloning for downloads seems to help. Unshallowing seems a bit slow with bigger projects (required when first downloading and then sync'ing a project), but that's unlikely to happen often.
Also fixed the "file not found" errors seen when downloading. Did not change the current behaviour of always downloading a zip of all translations.