EvalAI icon indicating copy to clipboard operation
EvalAI copied to clipboard

[#4670]Add bi-directional sync feature

Open Zahed-Riyaz opened this issue 4 months ago • 2 comments

This PR in tackling issue #4760, enables syncing Challenge and Challenge Phase updates made via the EvalAI UI to the configured GitHub repository/branch. When a supported field changes in the UI, a corresponding commit is made to the repo to keep configuration in sync.

EvalAI-Starters side : https://github.com/Cloud-CV/EvalAI-Starters/pull/129

How it works

  • Post-save hooks in apps/challenges/models.py:
    • challenge_details_sync for Challenge
    • challenge_phase_details_sync for ChallengePhase
  • Determines the changed field from update_fields or infers it from the request payload keys to make a minimal, single-field commit.
  • Calls challenges.github_utils sync helpers to apply changes on GitHub.

Safeguards

  • Skips sync for:
    • Bot-triggered saves (prevents recursion)
    • GitHub-sourced changes (prevents loops)
    • Repeated syncs within the same request (per-request dedupe)
  • Requires github_repository, github_branch, and github_token to be set on Challenge.

Configuration

  • Set on each Challenge:
    • github_repository (e.g., org/repo)
    • github_branch (defaults handled in code)
    • github_token (PAT with repo access)

Migrations

  • 0113_add_github_branch_field_and_unique_constraint.py now:
    • Adds github_branch if missing (SQL)
    • Records the field in Django state (no-op DB)
    • Adds a partial unique index on (github_repository, github_branch) when both are non-empty
  • Removed redundant 0116_challenge_github_branch.py so there’s a single source of truth.

Django signals

  • Receivers:
    • challenge_details_sync on Challenge (post_save)
    • challenge_phase_details_sync on ChallengePhase (post_save)
  • Behavior:
    • Trigger only on updates when github_repository and github_token are set.
    • Determine the changed field via update_fields or inferred request payload keys.
    • Commit a minimal, single-field change to GitHub through challenges.github_utils.
  • Guards:
    • Skip bot-triggered saves and GitHub-sourced changes to prevent recursion/loops.
    • Per-request dedupe avoids duplicate commits in the same request.

GitHubSyncMiddleware

  • Purpose: Resets per-request sync context and captures payload keys for changed-field inference.
  • How:
    • On POST/PUT/PATCH, extracts JSON or form keys into a thread-local store.
    • Works with the signal receivers to identify the precise field changed.
    • Prevents re-entrant syncs within the same request.

Dependencies

  • requests: Added/confirmed in requirements/common.txt to support GitHub API calls used by apps/challenges/github_interface.py and related utilities.

Zahed-Riyaz avatar Aug 11 '25 15:08 Zahed-Riyaz

Here is a video description of the feature : https://github.com/user-attachments/assets/f9701bf2-3729-4998-8ec3-9d060362965e

Zahed-Riyaz avatar Aug 30 '25 20:08 Zahed-Riyaz

Codecov Report

:x: Patch coverage is 56.81234% with 168 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 90.09%. Comparing base (3489634) to head (4a05fa5). :warning: Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
apps/challenges/github_interface.py 44.27% 112 Missing :warning:
apps/challenges/models.py 78.26% 25 Missing :warning:
apps/challenges/github_utils.py 64.91% 20 Missing :warning:
apps/challenges/github_sync_config.py 0.00% 5 Missing :warning:
apps/base/utils.py 20.00% 4 Missing :warning:
apps/challenges/serializers.py 66.66% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4765      +/-   ##
==========================================
- Coverage   91.90%   90.09%   -1.82%     
==========================================
  Files          85       88       +3     
  Lines        7142     7531     +389     
==========================================
+ Hits         6564     6785     +221     
- Misses        578      746     +168     
Flag Coverage Δ
backend 92.70% <56.81%> (-3.96%) :arrow_down:
frontend 87.25% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Accounts & Authentication 100.00% <ø> (ø)
Challenges Management 89.72% <57.29%> (-5.63%) :arrow_down:
Job Processing 98.19% <ø> (ø)
Participants & Teams 100.00% <ø> (ø)
Challenge Hosts 100.00% <ø> (ø)
Analytics 100.00% <ø> (ø)
Web Interface 100.00% <ø> (ø)
Frontend (Gulp) 87.25% <ø> (ø)
All Models 94.38% <78.26%> (-3.21%) :arrow_down:
All Views 100.00% <ø> (ø)
All Serializers 98.50% <66.66%> (-0.32%) :arrow_down:
Utility Functions 96.51% <20.00%> (-0.29%) :arrow_down:
Core Configuration 82.35% <ø> (ø)
Files with missing lines Coverage Δ
apps/challenges/urls.py 100.00% <ø> (ø)
apps/challenges/views.py 100.00% <ø> (ø)
apps/challenges/serializers.py 96.21% <66.66%> (-0.77%) :arrow_down:
apps/base/utils.py 95.74% <20.00%> (-2.79%) :arrow_down:
apps/challenges/github_sync_config.py 0.00% <0.00%> (ø)
apps/challenges/github_utils.py 64.91% <64.91%> (ø)
apps/challenges/models.py 91.33% <78.26%> (-4.49%) :arrow_down:
apps/challenges/github_interface.py 44.27% <44.27%> (ø)
Files with missing lines Coverage Δ
apps/challenges/urls.py 100.00% <ø> (ø)
apps/challenges/views.py 100.00% <ø> (ø)
apps/challenges/serializers.py 96.21% <66.66%> (-0.77%) :arrow_down:
apps/base/utils.py 95.74% <20.00%> (-2.79%) :arrow_down:
apps/challenges/github_sync_config.py 0.00% <0.00%> (ø)
apps/challenges/github_utils.py 64.91% <64.91%> (ø)
apps/challenges/models.py 91.33% <78.26%> (-4.49%) :arrow_down:
apps/challenges/github_interface.py 44.27% <44.27%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3489634...4a05fa5. Read the comment docs.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Sep 01 '25 20:09 codecov[bot]