argilla icon indicating copy to clipboard operation
argilla copied to clipboard

feat: add Python rq to execute background jobs

Open jfcalvo opened this issue 1 year ago • 1 comments

Description

This PR add the following changes:

  • [x] Add rq to help us execute background jobs.
  • [x] Add a background job to update all records for a dataset when the dataset distribution strategy is updated.
  • [x] Change HuggingFace Dockerfile to install Redis and run rq workers inside honcho Procfile.
  • [x] Add documentation about new ARGILLA_REDIS_URL environment variable.
  • [x] Add ping to Redis so Argilla server is not started if Redis is not ready.
  • [x] Change Argilla docker compose file to include a container with Redis and rq workers.
  • [x] Update Argilla server README.md file adding Redis as dependency to install.
  • [x] Add documentation about Redis being a new Argilla server dependency.
  • [ ] Remove some TODO sections before merging.
  • [ ] Allow tests job workers synchronously (with pytest) It's not working due to asyncio stuff (running an asynchronous loop inside another one, more info here: https://github.com/rq/rq/issues/1986).

Closes #5431

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested

  • [x] Testing it on HF Spaces.

Checklist

  • I added relevant documentation
  • I followed the style guidelines of this project
  • I did a self-review of my code
  • I made corresponding changes to the documentation
  • I confirm My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

jfcalvo avatar Aug 27 '24 09:08 jfcalvo

Codecov Report

Attention: Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.

Project coverage is 91.00%. Comparing base (fee1f5a) to head (96c96dc). Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
argilla-server/src/argilla_server/_app.py 27.27% 8 Missing :warning:
...lla-server/src/argilla_server/jobs/dataset_jobs.py 63.63% 8 Missing :warning:
argilla-server/src/argilla_server/cli/worker.py 55.55% 4 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5432      +/-   ##
===========================================
- Coverage    91.32%   91.00%   -0.33%     
===========================================
  Files          141      144       +3     
  Lines         5869     5915      +46     
===========================================
+ Hits          5360     5383      +23     
- Misses         509      532      +23     
Flag Coverage Δ
argilla-server 91.00% <63.63%> (-0.33%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 27 '24 10:08 codecov[bot]

The URL of the deployed environment for this PR is https://argilla-quickstart-pr-5432-ki24f765kq-no.a.run.app

github-actions[bot] avatar Sep 04 '24 12:09 github-actions[bot]