Avik Pal

Experimenting with Bumper.jl

A simple implementation: ```julia struct BroadcastLayer{T

https://github.com/FluxML/Optimisers.jl/pull/112 introduces support for frozen weights natively in Optimisers.jl. We still want to retain our current experimental API and just show how to use this native feature in a tutorial...

See https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.broyden2.html#scipy-optimize-broyden2 and https://web.archive.org/web/20161022015821/http://www.math.leidenuniv.nl/scripties/Rotten.pdf for rank-reduction. The currently implemented version corresponds to "simple". We would want to have the other variants as well.

- [ ] Accelerated DF-SANE https://arxiv.org/pdf/2104.13447.pdf - [ ] DF-SAUNE for Under-determined LS Problems https://www.scielo.br/j/cam/a/6nML5bXvthQ7nKRpFSjqyjH/?lang=en #345 sets up spectral methods in a nicer way but it would be nice to...

Add a wrapper for Optimization.jl

3

Unfortunately, this introduces a massive circular dependency if Optimization is loaded. But we only pay the price in CI rn.

Inexact Newton Steps in Levenberg Marquardt

2

See the last part of http://ceres-solver.org/nnls_solving.html#levenberg-marquardt. This is basically already there and this is mostly a internal solver selection mechanism: 1. Add a forcing function to LM 2. Add a...

- [ ] Add some tests - [ ] Overload `reinit` for that cache and remove Dual numbers before sending them to the inner caches

[WIP] Multi-Step Methods

1

## TODOs - [x] PotraPtak3 - [ ] SinghSharma - [ ] 4 - [ ] 5 - [ ] 7 - [ ] Tests - [ ] Documentation -...

Default Automatic Differentiation Choice

8

### Forward AD - [x] Non-Sparse Default AD should be `AutoPolyesterForwardDiff` (if that package is loaded). This will be similar to SimpleNonlinearSolve ### Reverse AD - For in-place problems, we...