pie
pie copied to clipboard
emanjavacas
Feature: allow picking whatever pytorch lr_scheduler
Here is a draft for the lr_scheduler params (related to #78) and how to avoid breaking backwards compatibility. Could you use it to test the results of the cosineannealing that you got?
From the last discussion,t we concluded to leave out the stuff coming from 3D party libraries (including the delayed stuff)
Delayer is not third party, I coded it :) And it's only with Delayer that Cosine outperforms significantly Adam :)
Could you pin point what is it that the Delayer class is achieving? I see a lot of new code involved in printing stuff and so on. Perhaps there is an easier way to add what the Delayer class contributes.
Ye, go ahead. Only I'd rather really keep it simple. For instance if there is that class that delays the step, we don't need to add a bunch of new functions to adapt the printing code for that class.
Any chance you could quickly replicate the results for the combination that was on par with ReduceLROnPlateau but was faster? (No rush here)
On Mon, Dec 7, 2020 at 6:25 PM Thibault Clérice [email protected] wrote:
Great !
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/emanjavacas/pie/pull/79#issuecomment-740062144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPIPI4DR6VURKQ7RURLHH3STUFYFANCNFSM4UQORVYQ .
-- Enrique Manjavacas
Running:
"lr_scheduler": "CosineAnnealingLR", // LR Scheduler to use: ReduceLROnPlateau,
"lr_scheduler_delay": 10,
"lr_scheduler_params": {
"T_max": 40,
"min_lr": 0.000001, // minimum learning rate
"lr_patience":7,
"factor":0.6,
},
It works: ran once on my personal GPU for 67 epochs (higher parts of the box in https://github.com/emanjavacas/pie/issues/76#issue-757884805) with score within the box 97.61. (previous version of comment was using dev score, because I am tired and can't read)