EvalAI

EvalAI copied to clipboard

Description

Currently, when someone creates a challenge on EvalAI, then gets the access to make submissions to that challenge in order to test it. But the catch here is that the submission worker is not ready to accept the submissions for that challenge since it needs to load the evaluation script and other required files. Hence whatever submission the challenge host does, it stays in the submitted state.

How to fix it

• [ ] Add support for loading the required challenge related files whenever the challenge is created making sure that submission worker is reloaded or is loading the required files in memory

Hey i'll try to work on this issue.

@deshraj @w3x10e8 I have fixed the issue. Can I submit my PR ?

@TejasBhitle hey sure you can submit your PR, Im working on other issue rn

Can I take up this issue ?

Sure @shiv6146, you can work on this.

@RishabhJain2018 Please have a look at #2205 I have made changes to submission_worker which did not have any unit tests previously and hence I think my coverage has reduced :disappointed: Any workarounds to actually fix this?

@shiv6146 This may help for now, until tests for the submission worker are written. Please have a look: https://coverage.readthedocs.io/en/coverage-4.3.4/excluding.html

What's the status of PR #2205?

I was not aware of the existing PR, and I created a simpler PR (2 line change plus comments) which addresses the issue: #3422.