create_translations: Enable background processing by default.

[mont29]

Make administrative commands like Update or Reset from the admin web UI reasonably responsive, by running the actual translation updates asynchronously in background tasks.

[!NOTE] This PR is mainly intended as a base for discussion. I do not have enough understanding of the Weblate/Celery projects to fully grasp the potential implications of the proposed changes.

Proposed changes

Adds a new run_async parameter to create_translations function, False by default.

When False, the update will processed immediately, and any failure will be reported.

When True (and the debug CELERY_TASK_ALWAYS_EAGER setting is not True), the update is send to a Celery task and will happen in the background.

The create_translations calls from add_new_language, do_update, do_reset, and do_file_scan are made asynchronous.

From a user PoV, it means that the execution of these tasks now returns (very) fast, and the actual processing happens in the background, which is trackable from the UI itself.

NOTE: This commit may also fix a theoretical re-entrant loop? If the call to create_translations from the Celery task failed again to acquire the component lock, existing code would add a new task?

Checklist

• [x] Lint and unit tests pass locally with my changes.
• [ ] I have added tests that prove my fix is effective or that my feature works.
• [ ] I have added documentation to describe my feature.
• [x] I have squashed my commits into logic units.
• [x] I have described the changes in the commit messages.

Unknowns and Open Questions

The main point I am unsure about this change is the behavior of Celery when submitting several times the same task.

From reading online, and some quick testing in the dev setup, it seems that adding the same task several times just does nothing, only the first one is executed?

Other information

Current weblate code gives a fairly weird and hard to understand behavior on admins commands that update translations, when the component is big (over 30k messages and over 50 languages). In our case (Blender UI and manual translations), an update can take up to over 2h in total.

This will always reach the timeout of the WSGI server - and in the over 2h cases, silently error when the process is terminated, leaving some languages unprocessed.

However, once in a while, the Update request will return successfully almost immediately, and then in the UI we get this nice clickable 'clock' icon showing that the background update task is running - with a nice 'update' page to follow the process even. The main target of this PR is to get that last behavior systematically.

[sentry[bot]]

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: weblate/trans/models/component.py

Function	Unhandled Issue
_create_translations	FileParseError: No such file or directory weblate... Event Count: 345
add_new_language	FileExistsError: [Errno 17] File exists: '/home/weblate/data/vcs/open-web-calendar/configuration-page-index-html/t... ... Event Count: 2

---

_{Did you find this useful? React with a 👍 or 👎}

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 61.70213% with 18 lines in your changes missing coverage. Please review.

Project coverage is 91.13%. Comparing base (10fea1e) to head (46dbd7c). Report is 2811 commits behind head on main.

Files with missing lines	Patch %	Lines
weblate/trans/models/component.py	52.63%	7 Missing and 2 partials :warning:
weblate/trans/views/git.py	57.14%	3 Missing and 3 partials :warning:
weblate/trans/views/basic.py	25.00%	2 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11717      +/-   ##
==========================================
- Coverage   91.14%   91.13%   -0.02%     
==========================================
  Files         586      586              
  Lines       60092    60111      +19     
  Branches     9505     9509       +4     
==========================================
+ Hits        54773    54782       +9     
- Misses       3688     3693       +5     
- Partials     1631     1636       +5     

Files with missing lines	Coverage Δ
weblate/addons/tasks.py	92.39% <100.00%> (ø)
weblate/glossary/tasks.py	79.31% <100.00%> (ø)
weblate/trans/models/translation.py	86.60% <100.00%> (ø)
weblate/trans/tasks.py	72.41% <100.00%> (ø)
weblate/trans/tests/test_git_views.py	100.00% <100.00%> (ø)
weblate/trans/views/basic.py	84.76% <25.00%> (-0.28%)	:arrow_down:
weblate/trans/views/git.py	68.23% <57.14%> (-3.39%)	:arrow_down:
weblate/trans/models/component.py	80.52% <52.63%> (-0.03%)	:arrow_down:

... and 1 file with indirect coverage changes

[nijel]

Generally, this is something we have to do in the long term to avoid overly long HTTP requests. The main reason for those being synchronous was to bring the user to the final state. And I think we need some UI improvements to see background operations progress immediately.

For example, when adding a new translation via add_new_language, the user is redirected to the newly created translation. But when the processing will be async, he ends up on empty translation, which is just being populated in the background. Yes, there is a way to watch this from the UI, but it's IMHO currently not reachable from the translation page.

A similar situation is with repository operations being asynchronous. Instead of seeing a new state like now, you would still see an old state and need to reload the page until it is updated.

I think the best approach would be to use JavaScript to submit the async operation (maybe using API, not a dedicated UI endpoints), immediately show the task progress and redirect once the task has been completed. We already do something similar with the automatic translation, but the code there doesn't redirect, it just shows the progress.

[mont29]

Thanks for the suggestions. I'll try to work on this asap :)

[mont29]

I added a redirect to the 'progress' page once git operations are done, for the Update/Reset/Rescan operations.

This is not (yet) change to use JS/API , but it also is easier to implement from existing code. :sweat_smile:

E.g. clicking on Reset redirects to this:

[mont29]

Think this PR is ready for final review now? Also updated it against latest main.

[mont29]

think this is ready for another pass of review now :)

[nijel]

I made some changes to the strings, but still would like to see a review from @orangesunny here.

[orangesunny]

I made some changes to the strings, but still would like to see a review from @orangesunny here.

I think you made it well and would use it. I only have some itching that Continue after completion misses the object and could work better like Continue <somewhere> after completion. But it probably not essential for understanding in the UI.