denizyuret

See if any low hanging fruits. https://discourse.julialang.org/t/ann-knet-1-4-0-accelerating-cuarrays/45206/18?u=denizyuret

Knet.Layers20

3

Copy of KnetLayers?

I started to port KnetLayers as Knet.Layers20 submodule. I will freeze it in its current state once the port is complete. @ekinakyurek can decide where to continue the development: e.g....

benchmarks with other dynamic frameworks

74

Let's benchmark and improve any shortcomings we find. We can track progress using this issue. Some resources: * [Dynet benchmarks](https://github.com/neulab/dynet-benchmark) * [Dynet paper](https://arxiv.org/pdf/1701.03980.pdf) * [Tensorflow Fold paper](https://openreview.net/pdf?id=ryrGawqex) * [Pytorch code](https://github.com/pytorch/pytorch)...

Use CUDA.jl functions where possible

5

Replacement functions and macros to use: (See https://juliagpu.gitlab.io/CUDA.jl/tutorials/introduction) * CUDA.@sync, synchronize() * CUDA.fill(1.0f0, N), CUDA.zeros(N), CUDA.ones(Int, N) * @cuda threads=256 gpu_kernel_in_julia!(y_d, x_d) * cudaconvert, cudacall low level fns used by...

- [ ] activation: add tests - [ ] dropout: make it threadsafe - [ ] multiheadattn - [ ] softmax - [ ] rnn - [ ] ctc -...

- [x] move atype to Knet.jl - [ ] move param to Layers21. - [x] implement Uniform(r) and short form U as init fn. Define fanin, fanout. Write the formula...