horde-ad

horde-ad copied to clipboard

Investigate HaskTorch

1

I assumed @hasktorch wraps the extensive PyTorch API in some very smart Haskell type-level programming. However, @arkadiuszbicz hypothesises that HaskTorch is using not PyTorch, but a CI library Torch that...

Mikolaj

Find a way to implement mapping and other higher order operations on dual numbers efficiently, despite function space not being a differentiable type

4

Edit2: this should be completely rewritten and the version in Google doc from 9 March is not coherent, either. For a start, please ignore this ticket description and see a...

Mikolaj

Implement bachnorm layer and add it to some tests

2

https://en.wikipedia.org/wiki/Batch_normalization https://penkovsky.com/neural-networks/day4/ I'm sure there are good blog posts about that. Looking this up in pytorch and/or tensorflow API may help, too. Perhaps adding this to the existing tests with...

Mikolaj

Implement LSTM with both soft and hard attention

2

Implements a couple of variants of LSTM. Then let's create some example neural networks with that (e.g., modify the existing MNIST RNN to use LSTM). If not and if no...

Mikolaj

Code an example with audio recordings of speech, e.g., sentiment analysis

7

Audio data is much smaller than video and probably also than photos, so there's a chance this fits CPU processing. Being ~linear~[edit: sequential], this may be a good fit for...

Mikolaj

document the core API of the Engine

1

Write nice Haddock for the following 3 functions in `HordeAd/Core/Engine.hs`: `dReverse`, `dForward`, `dFastForward`. Some inspiration can be drawn from `HordeAd/Internal/Delta.hs:gradientFromDelta`. Links to the code (ATM): https://github.com/Mikolaj/horde-ad/blob/91dc3840ee27cc75ce1b7c5a1411600eb66bfe94/src/HordeAd/Core/Engine.hs#L144 https://github.com/Mikolaj/horde-ad/blob/91dc3840ee27cc75ce1b7c5a1411600eb66bfe94/src/HordeAd/Core/Engine.hs#L180 https://github.com/Mikolaj/horde-ad/blob/91dc3840ee27cc75ce1b7c5a1411600eb66bfe94/src/HordeAd/Core/Engine.hs#L217 https://github.com/Mikolaj/horde-ad/blob/91dc3840ee27cc75ce1b7c5a1411600eb66bfe94/src/HordeAd/Internal/Delta.hs#L331

sfindeisen

Create bindings to libxsmm (in a separate package)

2

See https://github.com/Mikolaj/horde-ad/issues/9#issuecomment-1086211063. To test the bindings, implement convolution in horde-ad tests using them.

Mikolaj

How can we reduce type parameters further?

4

I think we need @goldfirere to help with this. To simplify the surface API we'd like to reduce the length number of type parameters floating around. Currently we have two...

tomjaguarpaw

investigate huge floating point errors in (flaky) tests

1

Depending on the architecture and/or platform, we're sometimes getting test errors like: ``` FAIL (31.18s) test/common/TestMnistFCNN.hs:132: expected: 0.8991 but got: 0.7424999999999999 Use -p '/2 artificial 5 4 3 2 1...

sfindeisen

If a gradient is not updated by eval, this is either an experiment ("what happens if I remove this expression") or a typo causing unused parameter. We can detect this...

Mikolaj