Meta-Weight-Net_Code-Optimization

Meta-Weight-Net_Code-Optimization copied to clipboard

Hi, thank you very much for the code. However, for noisy labels, I do not observe significant difference between MWN and normal training (without MWN) when the corruption prob is 0.4. Did you try this experiment as well? Looking forward to your reply.

Can you tell me more details of your experiment? I remember that the result of the experiment has never been lower than 86%.

Thank you very much for the prompt reply! Yes, indeed the experiment with MWN is 85.56(final) and 86.77(best), but if I remove the MWN reweighting, the performance degradation is not severe either, which is 84.30(final) and 86.53(best). I ran with the original setting in this implementation. For the baseline, I simply commented the MWN off and changed all the weights to 1.

It would be great if you can conduct the baseline experiment and let me know the results, just in case if there's something I've done incorrectly. Much appreciated! Your way of implementing MetaSGD is quite inspiring.

Sorry, I did not do a baseline experiment. According to the original paper, the baseline experiment result should be around 70%.

Can this be the version of pytorch?

Can you send me your baseline code?

The code is attached below. Thank you.

On Fri, Oct 22, 2021 at 11:00 AM ShiYunyi @.***> wrote:

Can you send me your baseline code?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949246813, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU2QY5O65UBIFT63WNDUIDHVHANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Best Regards, Renjie PI

Sorry, I didn't find it. Can you send it to my email ([email protected])?

Sure, I just sent it.

On Fri, Oct 22, 2021 at 12:09 PM ShiYunyi @.***> wrote:

Sorry, I didn't find it. Can you send it to my email @.***)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949270316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU4WSW6XROM2BD24AJ3UIDPYPANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Best Regards, Renjie PI

Yes, I did. setting it to true wil disable MWN

ShiYunyi @.***> 于 2021年10月22日周五 12:45写道：

Did you add --no_tune_score True? I tried it and the default setting is false.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949283339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU5OEZFSLUBX72XOJK3UIDT77ANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

just add --no_tune_score in command line will do the trick

Pi, Renjie @.***> 于 2021年10月22日周五 14:31写道：

Yes, I did. setting it to true wil disable MWN

ShiYunyi @.***> 于 2021年10月22日周五 12:45写道：

Did you add --no_tune_score True? I tried it and the default setting is false.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949283339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU5OEZFSLUBX72XOJK3UIDT77ANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I asked the original author of mwn. He said it was the pytorch version. In newer versions of pytorch, the model becomes robust to label noise.

Thank you very much. This is strange, so does it mean that this method is not as useful for current pytorch version? XD. Thanks for the information though!

On Fri, Oct 22, 2021 at 3:29 PM ShiYunyi @.***> wrote:

I asked the original author of mwn. He said it was the pytorch version. In newer versions of pytorch, the model becomes robust to label noise.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949362808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU5XIYFDDV7MKALCQEDUIEHDZANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Best Regards, Renjie PI

My collaborator just ran the original code without MWN using torch 0.4.0, it can still reach 86.5. But it's fine, the idea of the paper is interesting.

On Fri, Oct 22, 2021 at 3:51 PM Pi, Renjie @.***> wrote:

Thank you very much. This is strange, so does it mean that this method is not as useful for current pytorch version? XD. Thanks for the information though!

On Fri, Oct 22, 2021 at 3:29 PM ShiYunyi @.***> wrote:

I asked the original author of mwn. He said it was the pytorch version. In newer versions of pytorch, the model becomes robust to label noise.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-949362808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU5XIYFDDV7MKALCQEDUIEHDZANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Best Regards, Renjie PI

-- Best Regards, Renjie PI

This is very strange. Maybe I wrote a wrong data set. Please wait a few days. Let me modify the code and do some experiments. Thank you very much for finding this error.

Hi,I saw some closed issues from the original github page, which were asking the same question, but left unaddressed.

Does that mean that my code has no errors compared to the original author? I can’t find the bug recently.

yeah， probabaly this issue already existed in previous version.

ShiYunyi @.***> 于 2021年10月25日周一 16:46写道：

Does that mean that my code has no errors compared to the original author? I can’t find the bug recently.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ShiYunyi/Meta-Weight-Net_Code-Optimization/issues/1#issuecomment-950676982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU3TXOLDHLQRGCB4LLLUIUKPNANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Sure, I just sent it. … On Fri, Oct 22, 2021 at 12:09 PM ShiYunyi @.> wrote: Sorry, I didn't find it. Can you send it to my email @.)? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS6CU4WSW6XROM2BD24AJ3UIDPYPANCNFSM5GPMSCAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. -- Best Regards, Renjie PI

Can you please send your baseline code to my email( [email protected] ). This is a part of my graduation project and I'm eager to find the reason for this problem.

@pipilurj

Hi, I was reading this paper recently and saw your discussion above. I want to ask that if the "84.30(final)" you mentioned is the score at epoch=120?

Since in the fig.7(a) of the paper, the BaseModel can in fact do better than the proposed model (in the 80 epoch where the learing rate of BaseModel is divided by 10 for the first time). However, the author report the performance of BaseModel at epoch=120 and it degrade from 87% to 70% in [80,120]. So I doubt that "84.30(final)" isn't result at epoch=120 and is instead the result at some epoch near 80?