submission-criteria
submission-criteria copied to clipboard
numerai
Make the improvements suggested in improvements.pdf
The improvements suggested here seem like they would be very good. We're also interested in other improvements to the checks.
There is a 100 NMR bounty on this issue.
Hi Philip! The improvements doc was a fairly interesting read. Sounds like there are some fairly interesting things there I would like to get into. However I have a clarification - is the 100 NMR bounty for implementing/solving all of the described issues? Or would it be possible to submit a PR for solving only some of the issues (e.g. focusing on improving Concordance specifically)?
@isaykatsman The 100 NMR bounty is for all of the issues, however I'm open to splitting that up into specific parts of it. Can you suggest a reasonable division of tasks/bounty?
Alternatively, if you work together with someone to solve all the tasks, you could split the bounty between yourselves.
We need a quality benchmark before I'd feel comfortable pushing a large algorithmic change. To do this initially I used actual user submissions to calibrate things, but that would be hard to do.
For the third proposed concordance solution (use of the targets in the Test Set), how exactly would comparing the log loss of the validation and the test sets incur a data leak? Is it that users could modify only the test set predictions and based on the concordance outcome determine if their test set predictions are getting a better log loss?
