EvalAI icon indicating copy to clipboard operation
EvalAI copied to clipboard

Fix #3495: Change Token to Bearer

Open rcannood opened this issue 3 years ago • 6 comments

Fixes #3495.

Running eval.ai with docker-compose and trying to connect with a remote worker does not work. It seems the remote worker script is a bit out of date with the current API. The API documentation helped out a lot in figuring out what the issue was.

These are the issues I had to fix to get the script to work:

  • Rename Token to Bearer (L329).
  • Remove return statement (L527). I don't understand why this is there.
  • Allow changing the protocol for contacting the django server from http to https by setting export DJANGO_SERVER_PROTOCOL=https.
  • Fix the submission_data["result"] data structure format. The evaluation script is expected to return an object in the format of :
    [ 
      { "train-split" : { 
        "Metric" : 100 
      } } 
    ]
    
    but the API expects :
    [ 
      {
        "split": "train-split", 
        "show_to_participant" : True, 
        "accuracies" : { "Metric" : 100 } 
      }
    ]
    
    So a little bit of post-processing is needed. I could have done the postprocessing in the evaluation script itself, but I thought it'd be better to keep the same output format for the evaluation script so that switching between remote evaluation or not can still happen fairly quickly without needing to change the output format of the evaluation script.

rcannood avatar Jun 24 '21 11:06 rcannood

Codecov Report

Merging #3496 (ee504a5) into master (96968d6) will decrease coverage by 0.76%. The diff coverage is 34.17%.

@@            Coverage Diff             @@
##           master    #3496      +/-   ##
==========================================
- Coverage   72.93%   72.16%   -0.77%     
==========================================
  Files          83       20      -63     
  Lines        5368     3194    -2174     
==========================================
- Hits         3915     2305    -1610     
+ Misses       1453      889     -564     
Impacted Files Coverage Δ
frontend/src/js/controllers/authCtrl.js 53.91% <6.38%> (-12.95%) :arrow_down:
frontend/src/js/controllers/profileCtrl.js 79.76% <20.00%> (-13.10%) :arrow_down:
frontend/src/js/controllers/permissionCtrl.js 36.36% <22.22%> (-63.64%) :arrow_down:
frontend/src/js/controllers/challengeCtrl.js 64.68% <32.84%> (-9.02%) :arrow_down:
frontend/src/js/controllers/updateProfileCtrl.js 82.55% <44.44%> (-10.30%) :arrow_down:
frontend/src/js/controllers/challengeListCtrl.js 95.74% <50.00%> (+1.06%) :arrow_up:
...ntend/src/js/controllers/challengeHostTeamsCtrl.js 70.50% <66.66%> (-1.18%) :arrow_down:
frontend/src/js/controllers/teamsCtrl.js 71.17% <75.00%> (ø)
frontend/src/js/controllers/ChallengeInviteCtrl.js 100.00% <100.00%> (ø)
frontend/src/js/controllers/SubmissionFilesCtrl.js 95.45% <100.00%> (ø)
... and 30 more
Impacted Files Coverage Δ
frontend/src/js/controllers/authCtrl.js 53.91% <6.38%> (-12.95%) :arrow_down:
frontend/src/js/controllers/profileCtrl.js 79.76% <20.00%> (-13.10%) :arrow_down:
frontend/src/js/controllers/permissionCtrl.js 36.36% <22.22%> (-63.64%) :arrow_down:
frontend/src/js/controllers/challengeCtrl.js 64.68% <32.84%> (-9.02%) :arrow_down:
frontend/src/js/controllers/updateProfileCtrl.js 82.55% <44.44%> (-10.30%) :arrow_down:
frontend/src/js/controllers/challengeListCtrl.js 95.74% <50.00%> (+1.06%) :arrow_up:
...ntend/src/js/controllers/challengeHostTeamsCtrl.js 70.50% <66.66%> (-1.18%) :arrow_down:
frontend/src/js/controllers/teamsCtrl.js 71.17% <75.00%> (ø)
frontend/src/js/controllers/ChallengeInviteCtrl.js 100.00% <100.00%> (ø)
frontend/src/js/controllers/SubmissionFilesCtrl.js 95.45% <100.00%> (ø)
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 50df9a8...ee504a5. Read the comment docs.

codecov-commenter avatar Jun 24 '21 11:06 codecov-commenter

Thanks for taking your time to review this PR ^^

I just pushed the requested changes to the patch-1 branch.

rcannood avatar Jul 15 '21 10:07 rcannood

Hello @Ram81 !

I made the requested changes, which wasn't a lot. Would you have some time to take a look at it?

Thanks! :) Robrecht

rcannood avatar Jul 17 '21 21:07 rcannood

Kind reminder for @Ram81. Is there anything blocking this PR from being merged? :bow:

rcannood avatar Aug 17 '21 04:08 rcannood

Closing pull request as you don't seem to be interested anymore.

Let me know if you want want me to fix the remote worker.

rcannood avatar Sep 15 '21 10:09 rcannood

Hi @rcannood, sorry about the delay. I had missed this PR. I am reopening this PR as it has a bunch of changes that would help set up remote evaluation challenges easily. Thanks for taking time to work on this PR.

Ram81 avatar Dec 08 '21 17:12 Ram81

Closing this PR

rcannood avatar Jun 20 '23 08:06 rcannood