EvalAI icon indicating copy to clipboard operation
EvalAI copied to clipboard
verified publisher icon Cloud-CV

List the aaps, API's etc. to increase the codecov coverage

Open RishabhJain2018 opened this issue 9 months ago • 16 comments

The codecov coverage is at ~69% for now. Please list the apps, API's where it is missing the tests.

RishabhJain2018 avatar Mar 16 '25 05:03 RishabhJain2018

@RishabhJain2018 I would like to work on this issue.Can you please assign it to me ?

Akshat453 avatar Mar 16 '25 05:03 Akshat453

@RishabhJain2018 I would also like to contribute on this issue.

PrasadCodesML avatar Mar 16 '25 05:03 PrasadCodesML

Sure, @PrasadCodesML @Akshat453 You both can work together on this.

RishabhJain2018 avatar Mar 16 '25 05:03 RishabhJain2018

@RishabhJain2018 would like to work on this please

lazyperson1020 avatar Mar 16 '25 08:03 lazyperson1020 

53.92% (117/217) /code/frontend/src/js/controllers/authCtrl.js
 93.10% (54/58) /code/frontend/src/js/controllers/challengeCreate.js
 61.24% (1079/1762) /code/frontend/src/js/controllers/challengeCtrl.js
 70.51% (153/217) /code/frontend/src/js/controllers/challengeHostTeamsCtrl.js
100.00% (37/37) /code/frontend/src/js/controllers/ChallengeInviteCtrl.js
 80.72% (134/166) /code/frontend/src/js/controllers/analyticsCtrl.js
 92.42% (61/66) /code/frontend/src/js/controllers/challengeListCtrl.js
 99.07% (107/108) /code/frontend/src/js/controllers/dashCtrl.js
 68.99% (89/129) /code/frontend/src/js/controllers/featuredChallengeCtrl.js
100.00% (53/53) /code/frontend/src/js/controllers/contactUsCtrl.js
100.00% (57/57) /code/frontend/src/js/controllers/changePwdCtrl.js
100.00% (46/46) /code/frontend/src/js/controllers/mainCtrl.js
100.00% (39/39) /code/frontend/src/js/controllers/hostedChallengeCtrl.js
100.00% (26/26) /code/frontend/src/js/controllers/ourteamCtrl.js
 36.36% ( 8/22) /code/frontend/src/js/controllers/permissionCtrl.js
 71.18% (163/229) /code/frontend/src/js/controllers/teamsCtrl.js
 66.51% (141/212) /code/frontend/src/js/controllers/profileCtrl.js
 95.45% (21/22) /code/frontend/src/js/controllers/SubmissionFilesCtrl.js
 82.56% (71/86) /code/frontend/src/js/controllers/updateProfileCtrl.js
 90.91% (20/22) /code/frontend/src/js/controllers/webCtrl.js
Total Coverage:  69.28%

lazyperson1020 avatar Mar 16 '25 12:03 lazyperson1020

Hey, @RishabhJain2018 these are some tests which are missing in the codebase which will help improve codecov coverage, mostly the coverage in these folder is less authCtrl.js , challengeCtrl.js, permissionCtrl.js also i have observed that the test cases related to backend are present in the codebase in EvalAI\tests\integration and in EvalAI\tests\unit but in the codecov analysis there is only frontend coverage shown (~69%) it seems that the backend tests are not getting tested

Image

List of test cases are as follows

Backend Tests

  1. accounts/ Token Lifecycle Test: Verify token creation, expiration, and renewal for account verification Account State Management: Test transitions between unverified, verified, and disabled states Session Handling: Test concurrent session management and invalidation

  2. analytics/ Submission Pattern Analysis: Test analysis of submission patterns over time Performance Metrics: Test calculation of challenge performance metrics Data Aggregation: Test analytics data aggregation for dashboard displays

  3. challenges/ Phase Transition Logic: Test automatic transitions between challenge phases based on time Resource Quota Management: Test enforcement of submission limits per phase Challenge Dependencies: Test validation of inter-phase dependencies

  4. jobs/ Queue Priority System: Test job queue priority assignment and execution order Recovery Mechanism: Test system recovery after worker node failures Resource Allocation: Test distribution of jobs across worker nodes

  5. participants/ Team Management: Test team size restrictions and member role assignments Submission Quotas: Test enforcement of submission limits per team Access Control: Test team-based access restrictions to submissions

  6. worker/ Worker State Management: Test worker node state transitions Load Distribution: Test distribution of work across multiple workers Failure Recovery: Test worker recovery after system crashes

Frontend Tests

  1. web/ Challenge Phase UI: Test UI state updates during phase transitions Leaderboard Updates: Test real-time leaderboard data updates Error Display: Test error message display and formatting

  2. evalai/ API Response Handling: Test frontend handling of different API response codes State Management: Test UI state management during async operations

PrasadCodesML avatar Mar 16 '25 15:03 PrasadCodesML

@RishabhJain2018 Sir, I have found that backend coverage is about 37% when tested locally.

lazyperson1020 avatar Mar 16 '25 15:03 lazyperson1020

Hello @RishabhJain2018 ,Following are the APIs and backend apps that are currently missing test cases:

List of API Endpoints Missing Test Cases:

Auth & Accounts:

  • /auth/register/ (POST)
  • /auth/login/ (POST)
  • /accounts/user/get_auth_token (GET)
  • /accounts/user/refresh_auth_token (GET)
  • /auth/password/change/ (POST)
  • /accounts/user/resend_email_verification/ (POST)

Challenges:

  • /challenges/{challengeId}/manage_worker/{action}/ (PUT)
  • /challenges/{challengeId}/request_approval (GET)
  • /challenges/{challengeId}/get_worker_logs/ (GET)
  • /challenges/challenge/{challengeId}/prizes/ (GET)
  • /challenges/challenge/{challengeId}/sponsors/ (GET)
  • /challenges/{challengeId}/participate/ (POST)
  • /challenges/{challengeId}/scale_resources/ (PUT)
  • /challenges/challenge/{challengeId}/get_participant_teams (GET)
  • /challenges/challenge/{challengeId}/deregister/ (POST)

Jobs:

  • /jobs/phase_split/{phaseSplitId}/public_leaderboard_all_entries/ (GET)
  • /jobs/challenges/{challengeId}/submissions/{submissionId}/update_submission_meta/ (PATCH)

Backend Apps That Lack Sufficient Test Coverage:

Models:

  • apps/hosts/models.py
  • apps/jobs/models.py
  • apps/base/models.py
  • apps/participants/models.py
  • apps/accounts/models.py
  • apps/challenges/models.py

Serializers:

  • apps/challenges/serializers.py
  • apps/participants/serializers.py
  • apps/accounts/serializers.py
  • Views & Admin:
  • apps/web/views.py
  • apps/challenges/admin.py
  • apps/web/admin.py
  • apps/participants/admin.py
  • apps/jobs/admin.py
  • apps/hosts/admin.py

Utilities & Celery Tasks:

  • apps/base/utils.py
  • apps/jobs/utils.py
  • apps/jobs/tasks.py
  • apps/jobs/sender.py
  • apps/challenges/utils.py
  • apps/hosts/utils.py
  • apps/challenges/aws_utils.py
  • evalai/celery.py

Akshat453 avatar Mar 16 '25 15:03 Akshat453

Thanks @PrasadCodesML @Akshat453 for the detailed analysis. Let me go through it and will open issues for it.

RishabhJain2018 avatar Mar 16 '25 20:03 RishabhJain2018

@lazyperson1020 @PrasadCodesML @Akshat453 What percentage % of backend code already has test cases? Is there a similar breakdown for the backend?

gautamjajoo avatar Mar 17 '25 07:03 gautamjajoo

@gautamjajoo
Backend has 58% code coverage below images is when i ran the test locally

Backend

Image

Image

PrasadCodesML avatar Mar 17 '25 09:03 PrasadCodesML

@gautamjajoo some tests failed when I ran locally. Just wanted to ask is that an issue of the system? ( I am using mac m1)

=========== 48 failed, 536 passed, 93 warnings in 151.61s (0:02:31) ============

lazyperson1020 avatar Mar 17 '25 10:03 lazyperson1020

Any updates on this issue? Would like to work on increasing code coverage

Zahed-Riyaz avatar Mar 28 '25 10:03 Zahed-Riyaz

Hey @Zahed-Riyaz , Sure, please start working on it.

Note that, please open separate PR for each file. Also, this issue can be broken into multiple smaller issues. Please open a issue for which you are writing tests and open the PR for the same.

RishabhJain2018 avatar Mar 28 '25 14:03 RishabhJain2018

@RishabhJain2018 can I work on some of them ?

lazyperson1020 avatar Mar 28 '25 14:03 lazyperson1020

Note that, please open separate PR for each file. Also, this issue can be broken into multiple smaller issues. Please open a issue for which you are writing tests and open the PR for the same.

Noted! I'll open an issue when I'm ready to work on it

Zahed-Riyaz avatar Mar 28 '25 15:03 Zahed-Riyaz