fastMRI
fastMRI copied to clipboard
facebookresearch
Failed submissions to fastmri.org when using Google Drive
This is something we identified in Discussion #220. It seems Google has changed their API recently so that our old code will fail to download a submission. Until I get some time to fix this, please submit to fastmri.org using Dropbox or an S3 bucket.
Following that, it seems like submitting using Dropbox also causes issues. I made a test submission yesterday using Dropbox, and it appears that the whole public leaderboard went down. @mmuckley you may want to take a look at that.
Okay I'm reaching out to a contractor that helps us run the site. Will try to follow-up when I can get something.
Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.
EDIT: I may have spoken too soon.
Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.
EDIT: I may have spoken too soon.
Hi @mmuckley, I think it's my fault again for the site being down. I noticed it was working like yesterday at some point and tried to resubmit. Then it went down again.
All I do is to set a dropbox link and change the last digit (on the URL) from 0 to 1 to enable writing permissions. The zip file is of the form mymodel.zip -> mymodel/multicoil_test/ . Btw I submitted on the Multi-Coil Brain track.
Maybe I should wait until any further notice from you to resubmit.
Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.
Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.
Hi, I can see that the leaderboard is up and running now but still I cannot properly submit. I can make a new submission but it doesn't show up in the leaderboard, neither on my account. On my account all the submissions are gone.
I even made a new account and made a new submission. I could see the submission was processing (on my personal board) and I got the email that it was successful. But it gone completely and it doesn't show up on the leaderboard at all.
Thanks @wdika for your assistance. I was able to identify the issue and you should see everything now.
Great @mmuckley, thanks for the update! You might remove the last models on the leaderboard since they are failed (as expected), because I cannot see them on my account.
@mmuckley I can submit now and see my submission but I think the targets have disappeared and the metrics are off. At first, I thought my reconstructions were faulty but as I realize now the targets are not there (left side). Can you please verify this? You may check the last model (xyz) on the brains leaderboard.
@wdika it looks like your images might be on the wrong scale?
The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.
@wdika it looks like your images might be on the wrong scale?
The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.
I made a few submissions, where I tested the scaling but none worked. Of course, it doesn't make sense to touch the evaluation code, but I am wondering why the targets' images show all black and the reconstructions look okay. Is this behavior expected?
No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.
Sorry - don't think I can help more with this; it's not a bug with the site.
No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.
Sorry - don't think I can help more with this; it's not a bug with the site.
Yeah, this is what I do. Thanks for all your help!
Resolved comments unrelated to issue title. If continued problems arise, please create a new issue.
Hello @Can-Zhao it is not resolved, mostly because I'm working on several other projects and don't have time to fix it. I apologize for the inconvenience. Using an S3 bucket or Dropbox should still work.