Kyle Daruwalla

Now that #16 is merged, we could support higher order rules if someone wanted to write them.

> While going through PS, I felt there was some extra boilerplate needed for simple utilities, so I want to keep the API minimal. I think it's fair to say...

Also mentioned on Slack, how do implement time step state consistently for a group of parameters? There is no need to worry about this. As long as we define `init(s::Schedule,...

> What would you have apply do? My point exactly. It seems weird to me that some structs in Optimisers.jl will implement `init`/`apply` and some will only implement `init`. My...

> I am trying to steer the conversation into finding out what building blocks we need to support to write proper scheduling routines. > > This isn't about PS and...

Thinking about this a little more and trying to be more concrete about my thoughts (TL;DR - merge `Schedule` after review, don't write schedule policies like `Inv` in Optimisers.jl): First,...

> I think it might be reasonable to move scheduling functions from PS into here (or a separate Schedulers.jl) Or just keep them in PS and use them here and...

> I am just suggesting that a lot of completed work is being redone. Have you seen the darsnack/rm-optim branch of PS? It incorporates almost all of your suggestions, and...

> The reason I want to hook in to the update (and therefore apply) is that that allows people to pass in a scheduled optimiser without changing any code in...

> scheduling functions are infinite (as long as they are functions) Not just arbitrary functions. Almost all common schedules like exponential decay, inverse decay, cosine annealing, cyclic LR, etc. are...