EvalAI icon indicating copy to clipboard operation
EvalAI copied to clipboard

Remote evaluation doesn't work

Open rcannood opened this issue 3 years ago • 0 comments

Running eval.ai with docker-compose and trying to connect with a remote worker does not work. It seems the remote worker script is a bit out of date with the current API. The API documentation helped out a lot in figuring out what the issue was.

These are the issues I had to fix to get the script to work:

  • Rename Token to Bearer (L329).
  • Remove return statement (L527). I don't understand why this is there.
  • Fix the submission_data["result"] data structure format. The evaluation script is expected to return an object in the format of :
    [ 
      { "train-split" : { 
        "Metric" : 100 
      } } 
    ]
    
    but the API expects :
    [ 
      {
        "split": "train-split", 
        "show_to_participant" : True, 
        "accuracies" : { "Metric" : 100 } 
      }
    ]
    
    So a little bit of post-processing is needed.

I will update #3496 to contain these fixes.

@Ram81: After I update the PR, can you review my changes?

rcannood avatar Jun 24 '21 11:06 rcannood