2nd Order Optimi z/s ers

[samiit]

Hi, I am new to this ecosystem and was exposed to it via the talk by Dhairya at JuliaCon2020. I thought to ask if there are any plans for you to include 2nd order Optimizers, that were recently introduced by the very recent work on AdaHessian. Kindly share if it is already included and if there are any benchmarks as to how they perform compared to the 1st order optimizers! Thanks.

Sam

[freemin7]

Another optimizer that uses higher order information is called Ridge Rider which might be really interesting to implement

[darsnack]

Now that #16 is merged, we could support higher order rules if someone wanted to write them.