ReScience-submission
ReScience-submission copied to clipboard
ReScience
Use binder for notebooks
mybinder.org lets people run jupyter notebooks from github repos in the cloud for free -- it might significantly decrease install issues for reviewers.
I have some experience with mybinder.org. The "regular" service seems to have been unreliable for quite some time, and only the beta service https://beta.mybinder.org/ seems to be working right now.
While having a version of the code as a mybinder notebook is certainly a good thing to disseminate the research to a broad audience, it is not necessarily the best way to create a robust method to make the code run over the medium or long-term.
mybinder notebooks should be encouraged, but a more regular and resilient way to install the software should probably be required.
@tpoisot Do you suggest author to prepare a notebook for each submission (when possible)?
At least offering this as a possibility. This way, if the reviewers don't feel like re-installing the entire environment, they can run the code in the cloud.
There are good reasons for propsing this option, but there is also one good argument against it: reviewers using binder will no longer check the installation instructions.
What about repo2docker? People can confirm both that installation works and build images locally. crowdAI does this for their competition submissions:
https://github.com/crowdAI/crowdai-musical-genre-recognition-starter-kit/blob/master/Round2_packaging_guidelines.md
It would be nice to do a real-life test with such an approach. Ask a few submitting authors if they are willing to play guinea pig and use repo2docker. Then see how reviewers react. Note that verifying the local installation is a task mostly unrelated to the rest of the review, and thus could be delegated to someone else, even someone with a different scientific background (or none at all).