Avik Pal

Slow Broadcasting (compared to Zygote)

4

```julia using NNlib, Enzyme, Zygote gelu_act(x) = sum(abs2, gelu.(x)) x = randn(Float32, 32, 32, 1, 32) @btime Enzyme.gradient(Reverse, $gelu_act, $x); # 1.298 ms (65 allocations: 386.95 KiB) @btime Zygote.gradient($gelu_act, $x);...

Minimal CUDA version for Lux Dense Layer

8

```julia using CUDA, Enzyme w = cu(rand(Float32, 10, 10)) b = cu(rand(Float32, 10)) x = cu(rand(Float32, 10, 10)) act = x -> max(0f0, x) function loss_function(act, w, x, b) return...

This can be used to store the timer_outputs in the solution object, and we don't need to forcefully use the caching API in https://docs.sciml.ai/NonlinearSolve/stable/basics/diagnostics_api/#Example-Usage.

NonlinearProblem is not a DEProblem

2

Fixes https://github.com/SciML/NonlinearSolve.jl/issues/154 A bit more involved than I thought :cry:

[WIP] Testing out a `vmap` implementation

1

## Example Usage Package is private currently ```julia using AutoBatching, SimpleNonlinearSolve function solve_nlprob(f::F, u0::AbstractVector, p::Number, alg) where {F} prob = NonlinearProblem(f, u0, p) return solve(prob, alg).u end u0 = rand(3)...

[DO NOT MERGE] Testing the new NonlinearSolveBase.jl

1

This is VERY Breaking * Move to NonlinearSolveBase, which is a significantly lighter version of DiffEqBase * ForwardDiff and FiniteDiff are moved into extensions, so no AD package is loaded...

[WIP] Simple JFNK

1

I was playing around with JNFK for DEQs and the Nonlinear Solve implementation is a bit too complicated for those purposes - [x] Working Version - [ ] Batched Version...

Add a tutorial on using SimpleNonlinearSolve inside kernels

4

We have an example of this in the paper, we just need to write it up as a tutorial.

**Is your feature request related to a problem?** Special Trust Region Algorithm for NLLS Problems with bound constraints. **Describe the solution you’d like** See https://nmayorov.wordpress.com/2015/06/19/trust-region-reflective-algorithm/ for a description of the...