Carlo Lucibello

@DilumAluthge there is something I don´t understand, from the answer to [this question](https://github.com/JuliaRegistries/General/pull/32119#issuecomment-816976264), the relocation process seemed quite easy...

ok, that seems something to consider. Let´s wait for some feedback from @vchuravy and @jpsamaroo

it seems I'm the only one hyped by this subdir approach, let's move everything to the separate repo https://github.com/FluxML/NNlibCUDA.jl then. I'll make a PR removing NNlibCUDA from here soon

reopening this after a few months of working with NNlibCUDA as a separate package. In the work for gather/scatter we add to ping-pong between the two libraries multiple times, it...

> I am wondering if NNlib should remove its own implementation and just reexport StatsFuns.logsumexp? I guess we should. I'm just wary of adding another dependency, there have been already...

LogExpFunctions.jl should define the rrules. We could do it here, but the original repo is the natural place. Also, if we need this, we'll need to define sepate implementations for...

> FYI recently I added the ChainRules definitions to LogExpFunctions. Not something that we typically pay much attention to (although we should!), but the rules themselves are differentiable?

I think we can avoid defining a mutating version `sparsemax!` and just go with `sparsemax`, since the implementation of `sparsemax!` does a lot of allocations in any case. Let's also...

I was looking into porting the functional form of normalization layers here, but I'm not sure how to handle the `Zygote.ignore` [block](https://github.com/FluxML/Flux.jl/blob/06970a5fbbb1cb485c5d2cba597a78fb453fc713/src/layers/normalise.jl#L180) without having NNlib depend on Zygote

could you update the OP with the benchmark result for your system?