dfdx
dfdx copied to clipboard
coreylowman
Deep learning in Rust, with shape checked tensors and neural networks
Related to #590 In my quest to speed up my language model training, I've been looking to implement Flash Attention (https://github.com/HazyResearch/flash-attention). They report linear storage requirements (still quadratic time requirements)...
Adds `SerializeWithModel` trait to allow serializing Gradients and Optimizers to files. Resolves #785, but technically will not implement SaveToNpz/LoadForNpz for optimizers, because we need an extra `model` parameter to structure...
Might be awesome to have inference in browser some day built with dfdx.
On long training runs it's required to checkpoint models in case something goes wrong part way through. When only saving the model, and resetting the optimizer state, training takes much...
#793 is prerequsite for this, closes #773 There's a very ugly hack with `fixup_batchnorms` that won't be necessary once #485 is resolved. There's also some type-level voodoo in `block.rs` to...
Often, it is recommend to loop over items in a cuda kernel like: ```c++ for (unsigned int i = tid; i < n; i += blockDim.x * gridDim.x) { ......
With #767 now added, we can integration test against more complex image networks! This new test should mirror the existing resnet18 integration test, but with a mobile net structure.
I'm in the process of implementing MobilenetV3 (and willing to contribute for integration tests), for that I need hard swish and hard sigmoid ops.
How would I create a kernel with a non-square/rectangular shape? Something like: ``` Conv2D ```
Metadata
Owner
Metadata
Deep learning in Rust, with shape checked tensors and neural networks