hoangmit

All the alternatives of the LATERAL clause should work. However, the first one failed as it cannot capture the variable. The 2nd version works mysteriously. ``` SELECT * FROM (...

Support pure pytorch implementation for memory_efficient_attention

4

# 🚀 Feature I found that `memory_efficient_attention` op does not support pure pytorch implementation (e.g. without using device specific op, or library or cython). The current implementations fail to dispatch...

Without "static" quantifier, it fails to compile in clang ``` ld.lld: error: undefined symbol: packNibbles >>> referenced by ggml.c:520 (llama_cpp/ggml.c:520) >>> .../llama_cpp/__ggml__/__objects__/ggml.c.pic.o:(quantize_row_q4_0) ld.lld: error: undefined symbol: bytesFromNibbles >>> referenced by...

Use RMSNorm

16

The original paper, and the reference implementation [1] uses RMS norm. However, llama.cpp uses ggml_norm() which looks like Layer norm? The differences between these may not be too obvious, because...

VeLO: Training Versatile Learned Optimizers by Scaling Up

2

Do we have any pytorch port of these learned optimizers ? Link: https://arxiv.org/abs/2211.09760v1

Slow

4

It is not a serious problem, but Bamtools is about 20% slower than the Python version aka pysam.

# 🚀 Feature Can we support this in xformers? First proposed in * https://arxiv.org/abs/2312.04927 * Mentioned in https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-based * and in https://hazyresearch.stanford.edu/blog/2024-03-03-based Size trade off version: https://arxiv.org/abs/2402.04347 Another variant: https://arxiv.org/abs/2402.10644...