pytorch-lr-finder
pytorch-lr-finder copied to clipboard
davidtvs
Steepest gradiant value
i want to be able to pull the value of steepest gradiant and use it in my code as a integer value. i see in the code it is under lr_finder.plot() but i am not abke to just assign min_grad to anything can you please help me
Hi @snoozeyouloose.
lr_finder.plot() will return a matplotlib.axes.Axes object and a suggested learning rate (which is currently determined by a point with negative & minimal gradient) when given argument suggest_lr is True.
Although you can't get the gradient of that point directly, you can still get it by performing the operation written in line 505 since history of learning rates and losses are all recorded in the lr_finder.history dictionary:
https://github.com/davidtvs/pytorch-lr-finder/blob/acc5e7ee7711a460bf3e1cc5c5f05575ba1e1b4b/torch_lr_finder/lr_finder.py#L500-L509
And I'm a bit curious about why you are trying to get the gradient of that point and using it as an integer. Because it's usually not necessary to know the gradient of that point, and it's also a bit weird to cast it (a float number) into an integer.
If you can provide more context of this problem, I am willing to dicuss it further with you.
I don't necessarily want to cast the steepest gradient. I want to use the best possible learning rate n use that in my code.. i m trying to run a program that iterativly finds the best loss over the best learning rate with variable number of hidden layer from 1 to 7 and 10 variable seeds. I m stuck at 2 placed first is the lrfinder best learning rate.. n R2score for each model..
If you have any suggestions please let me know
On Wed, 25 Nov 2020, 8:13 am Nale Raphael [email protected] wrote:
Hi @snoozeyouloose https://github.com/snoozeyouloose.
lr_finder.plot() will return a matplotlib.axes.Axes object and a suggested learning rate (which is currently determined by a point with negative & minimal gradient) when given argument suggest_lr is True.
Although you can't get the gradient of that point directly, you can still get it by performing the operation written in line 505 since history of learning rates and losses are all recorded in the lr_finder.history dictionary:
https://github.com/davidtvs/pytorch-lr-finder/blob/acc5e7ee7711a460bf3e1cc5c5f05575ba1e1b4b/torch_lr_finder/lr_finder.py#L500-L509
And I'm a bit curious about why you are trying to get the gradient of that point and using it as an integer. Because it's usually not necessary to know the gradient of that point, and it's also a bit weird to cast it (a float number) into an integer.
If you can provide more context of this problem, I am willing to dicuss it further with you.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/davidtvs/pytorch-lr-finder/issues/70#issuecomment-733514738, or unsubscribe https://github.com/notifications/unsubscribe-auth/APFLIGPQ2MEA3W47NMA6ZIDSRSVCLANCNFSM4UBUL7CA .
With regard to the problem of finding the best learning rate, you can just set the argument suggest_lr of lr_finder.plot() to True, then you can get the suggested value without recalculating it by yourself. As it stated in my previous comment, that suggested learning rate is currently determined by the steepest gradient, so it should meet your needs.
And if you want to prevent program getting stuck by the popup window resulted by plt.show() in lr_finder.plot(), you can pre-create a matplotlib.axex.Axes object and assign it to the argument ax of lr_finder.plot(). See also this example written by David: https://github.com/davidtvs/pytorch-lr-finder/pull/23#issuecomment-602248697
However, there are some reasons that we don't call the suggested learning rate as best learning rate, you may want to check out this thread for more details.
As for the R2 score for each model, I think it can be easily calculated by using sklearn.metrics.r2_score() after those models are well trained.
If you have any further questions, please just feel free to let me know!
