argilla
argilla copied to clipboard
feat: add Python rq to execute background jobs
Description
This PR add the following changes:
- [x] Add
rqto help us execute background jobs. - [x] Add a background job to update all records for a dataset when the dataset distribution strategy is updated.
- [x] Change HuggingFace Dockerfile to install Redis and run
rqworkers inside honcho Procfile. - [x] Add documentation about new
ARGILLA_REDIS_URLenvironment variable. - [x] Add ping to Redis so Argilla server is not started if Redis is not ready.
- [x] Change Argilla docker compose file to include a container with Redis and rq workers.
- [x] Update Argilla server
README.mdfile adding Redis as dependency to install. - [x] Add documentation about Redis being a new Argilla server dependency.
- [ ] Remove some
TODOsections before merging. - [ ]
Allow tests job workers synchronously (with pytest)It's not working due to asyncio stuff (running an asynchronous loop inside another one, more info here: https://github.com/rq/rq/issues/1986).
Closes #5431
Type of change
- New feature (non-breaking change which adds functionality)
How Has This Been Tested
- [x] Testing it on HF Spaces.
Checklist
- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature works
- I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
Codecov Report
Attention: Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.
Project coverage is 91.00%. Comparing base (
fee1f5a) to head (96c96dc). Report is 1 commits behind head on develop.
Additional details and impacted files
@@ Coverage Diff @@
## develop #5432 +/- ##
===========================================
- Coverage 91.32% 91.00% -0.33%
===========================================
Files 141 144 +3
Lines 5869 5915 +46
===========================================
+ Hits 5360 5383 +23
- Misses 509 532 +23
| Flag | Coverage Δ | |
|---|---|---|
| argilla-server | 91.00% <63.63%> (-0.33%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
The URL of the deployed environment for this PR is https://argilla-quickstart-pr-5432-ki24f765kq-no.a.run.app