ocannl

ocannl copied to clipboard

Implement "parameter punning" for the code notation `%cd`

1

This is really for show-off purposes. But it's just too tempting to not use it in the upcoming OCaml Workshop paper, and it adds extra consistency with the `%op` notation.

lukstafi

Study: [A Multi-Level Superoptimizer for Tensor Programs](https://arxiv.org/abs/2405.05751) see also: [Equality Saturation for Tensor Graph Superoptimization](https://arxiv.org/abs/2101.01332) (Found via Luminal Discord chat.)

lukstafi

Implement an xLSTM LLM

1

https://arxiv.org/abs/2405.04517

lukstafi

They put a load on the CI. - Highest priority: `ppx_jane` pulls in a lot of packages. - Medium priority: we don't use much from `core`. - Low priority: get...

lukstafi

Move the `%cd` syntax to the arrayjit library / package

1

This would go a long way in making arrayjit useful independently of OCANNL. Some complications might come from interactions with shape inference. But the biggest obstacle is that `%cd` also...

lukstafi

In my recent work I wasn't paying attention to dimension labels, only the dimensions (axis sizes) themselves. I suspect that they are not completely checked nor inferred.

lukstafi

Study and incorporate Andrej Karpathy's `llm.c` lessons

3

["A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch"](https://twitter.com/karpathy/status/1778988957713477778) https://github.com/karpathy/llm.c

lukstafi

Audit all places where that would be needed.

lukstafi

Add an LLVM / clang backend

1

Now no need to wait for LLVM 17: [bindings to LLVM 15 for OCaml 5 on opam](https://discuss.ocaml.org/t/ann-llvm-15-is-out/13019).

lukstafi

In the current giant refactor, I'm removing dynamic indexing. Also perhaps simplifying the "local / global on device / on host" behavior.

lukstafi