EvalAI
EvalAI copied to clipboard
Remote evaluation doesn't work
Running eval.ai with docker-compose and trying to connect with a remote worker does not work. It seems the remote worker script is a bit out of date with the current API. The API documentation helped out a lot in figuring out what the issue was.
These are the issues I had to fix to get the script to work:
- Rename
Token
toBearer
(L329). - Remove return statement (L527). I don't understand why this is there.
- Fix the
submission_data["result"]
data structure format. The evaluation script is expected to return an object in the format of :
but the API expects :[ { "train-split" : { "Metric" : 100 } } ]
So a little bit of post-processing is needed.[ { "split": "train-split", "show_to_participant" : True, "accuracies" : { "Metric" : 100 } } ]
I will update #3496 to contain these fixes.
@Ram81: After I update the PR, can you review my changes?