horde-ad
horde-ad copied to clipboard
Mikolaj
Higher Order Reverse Derivatives Efficiently - Automatic Differentiation library based on the paper "Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation"
I'm now leaning towards replicating the trick we have in Delta terms (Tom's Tree0 method --- a stamp per each term node). Then we could add the same stamp to...
It should be such that `tupdate (tzero sh) ix v` is the transpose of `tindex v ix`. Also https://github.com/Mikolaj/horde-ad/blob/6f88617de23a4d9fb328b352cf43fcf4cffd97b8/simplified/HordeAd/Core/AstSimplify.hs#L433 Probably `tscatter` can then be simplified using `tupdate` similarly as `tgather`...
Here's the Tom's idea from https://github.com/Mikolaj/horde-ad/issues/95#issuecomment-1503013877: > > the transposition can't be expressed as rewriting rules, because it's stateful. > > It's only stateful because it's Cayley-transformed. The eval function:...
That would happen whenever a `Tensor` class code is instantiated to the `Ast` instance, in the same step that vectorization happens (I'm not sure how these two would interleave). This...
Once a program is fully vectorized and so no higher order operations remain except `gather`, we should run a fusion transformation. However, the fusion should never create new higher order...
It turns out our handling of `AstAppend` was broken https://github.com/Mikolaj/horde-ad/commit/d3ddcf76572f49406a5d729570c6424b88f391c0#diff-89b28a3b50641a4f5bea4f760c491a369d1245c655cb471abec544c7438bc90dR312-R316 and because in our system append can't be expressed as gather nor build, in desperation we now use rule (D)...
Currently we just use vectors for the scaling factor in `eval1`, which is too dense and so not optimal, `MatrixOuter` for `eval2`, which is ad hoc, and tensors for `evalX`...
This is not urgent and, in fact, it's likely to complicate the code a lot without any benefit for neural network applications. Revisit when we have convincing other applications where...
Once vectorization on strictly regular array expressions works (these are expressions that result in regular arrays even after vectorization), generalize it to irregular arrays. Optionally, at the end, check regularity...
For any rank higher than 0, literals such as `42` have ambiguous shape and in ranked tensors this ambiguity usually can't be resolved. Branch https://github.com/Mikolaj/horde-ad/tree/ban-ambiguous-shape bans such cases (actually rank...