EvalAI
EvalAI copied to clipboard
Fix #3495: Change Token to Bearer
Fixes #3495.
Running eval.ai with docker-compose and trying to connect with a remote worker does not work. It seems the remote worker script is a bit out of date with the current API. The API documentation helped out a lot in figuring out what the issue was.
These are the issues I had to fix to get the script to work:
- Rename
Token
toBearer
(L329). - Remove return statement (L527). I don't understand why this is there.
- Allow changing the protocol for contacting the django server from http to https by setting
export DJANGO_SERVER_PROTOCOL=https
. - Fix the
submission_data["result"]
data structure format. The evaluation script is expected to return an object in the format of :
but the API expects :[ { "train-split" : { "Metric" : 100 } } ]
So a little bit of post-processing is needed. I could have done the postprocessing in the evaluation script itself, but I thought it'd be better to keep the same output format for the evaluation script so that switching between remote evaluation or not can still happen fairly quickly without needing to change the output format of the evaluation script.[ { "split": "train-split", "show_to_participant" : True, "accuracies" : { "Metric" : 100 } } ]
Codecov Report
Merging #3496 (ee504a5) into master (96968d6) will decrease coverage by
0.76%
. The diff coverage is34.17%
.
@@ Coverage Diff @@
## master #3496 +/- ##
==========================================
- Coverage 72.93% 72.16% -0.77%
==========================================
Files 83 20 -63
Lines 5368 3194 -2174
==========================================
- Hits 3915 2305 -1610
+ Misses 1453 889 -564
Impacted Files | Coverage Δ | |
---|---|---|
frontend/src/js/controllers/authCtrl.js | 53.91% <6.38%> (-12.95%) |
:arrow_down: |
frontend/src/js/controllers/profileCtrl.js | 79.76% <20.00%> (-13.10%) |
:arrow_down: |
frontend/src/js/controllers/permissionCtrl.js | 36.36% <22.22%> (-63.64%) |
:arrow_down: |
frontend/src/js/controllers/challengeCtrl.js | 64.68% <32.84%> (-9.02%) |
:arrow_down: |
frontend/src/js/controllers/updateProfileCtrl.js | 82.55% <44.44%> (-10.30%) |
:arrow_down: |
frontend/src/js/controllers/challengeListCtrl.js | 95.74% <50.00%> (+1.06%) |
:arrow_up: |
...ntend/src/js/controllers/challengeHostTeamsCtrl.js | 70.50% <66.66%> (-1.18%) |
:arrow_down: |
frontend/src/js/controllers/teamsCtrl.js | 71.17% <75.00%> (ø) |
|
frontend/src/js/controllers/ChallengeInviteCtrl.js | 100.00% <100.00%> (ø) |
|
frontend/src/js/controllers/SubmissionFilesCtrl.js | 95.45% <100.00%> (ø) |
|
... and 30 more |
Impacted Files | Coverage Δ | |
---|---|---|
frontend/src/js/controllers/authCtrl.js | 53.91% <6.38%> (-12.95%) |
:arrow_down: |
frontend/src/js/controllers/profileCtrl.js | 79.76% <20.00%> (-13.10%) |
:arrow_down: |
frontend/src/js/controllers/permissionCtrl.js | 36.36% <22.22%> (-63.64%) |
:arrow_down: |
frontend/src/js/controllers/challengeCtrl.js | 64.68% <32.84%> (-9.02%) |
:arrow_down: |
frontend/src/js/controllers/updateProfileCtrl.js | 82.55% <44.44%> (-10.30%) |
:arrow_down: |
frontend/src/js/controllers/challengeListCtrl.js | 95.74% <50.00%> (+1.06%) |
:arrow_up: |
...ntend/src/js/controllers/challengeHostTeamsCtrl.js | 70.50% <66.66%> (-1.18%) |
:arrow_down: |
frontend/src/js/controllers/teamsCtrl.js | 71.17% <75.00%> (ø) |
|
frontend/src/js/controllers/ChallengeInviteCtrl.js | 100.00% <100.00%> (ø) |
|
frontend/src/js/controllers/SubmissionFilesCtrl.js | 95.45% <100.00%> (ø) |
|
... and 30 more |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 50df9a8...ee504a5. Read the comment docs.
Thanks for taking your time to review this PR ^^
I just pushed the requested changes to the patch-1 branch.
Hello @Ram81 !
I made the requested changes, which wasn't a lot. Would you have some time to take a look at it?
Thanks! :) Robrecht
Kind reminder for @Ram81. Is there anything blocking this PR from being merged? :bow:
Closing pull request as you don't seem to be interested anymore.
Let me know if you want want me to fix the remote worker.
Hi @rcannood, sorry about the delay. I had missed this PR. I am reopening this PR as it has a bunch of changes that would help set up remote evaluation challenges easily. Thanks for taking time to work on this PR.
Closing this PR