Horace He

Alternate implementation of Lora leveraging tensor subclasses and reparametrization.

3

I thought this might be interesting as an alternate implementation of LoRA leveraging tensor subclasses and reparametrization. https://gist.github.com/Chillee/a8d2070b1b7b3f97d8c87bac3c366f8e The main idea here is that we can leverage parametrization in order...

Performance compared to local-attention + masking

9

I saw https://github.com/SHI-Labs/NATTEN/issues/89 > As far as I know both FAv2 and xFormers' FMHA support 1-D sliding window attention with causal masking, so you probably can use them for now,...

Rethink keybindings

4

I think we should rethink keybindings. I've always thought the huge package.json thing where there's a bunch of arbitrary conditions on each key was a bit of a mess. I...

I think that's the way to go. URI's can define custom read files and such.

The unit tests for VSCodeVim were awesome. @rebornix mentioned [spectron](https://electron.atom.io/spectron/) as a possibility? One other thing we might want is performance testing.

Make highlights look better

1

There's a couple ideas we can do for this. 1. I think we should try our best to use whatever highlights Vim provides, possibly? We can set a bunch of...

If you have a file like ``` jjjjjj | ``` and you have `imap jj `, autocompleting the `jjjjj` will cause you to leave insert mode, and not have the...

Add support for multiple flexattention calls in a single compile

1

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom): * __->__ #125516 * #125487 cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames...

[example] Added (hacky) Grok1 support

3

Downloading from https://huggingface.co/hpcai-tech/grok-1 ``` git clone --branch grok1 [email protected]:pytorch-labs/gpt-fast.git && cd gpt-fast/mixtral-moe export MODEL_REPO=hpcai-tech/grok-1 python scripts/download.py --repo_id $MODEL_REPO python scripts/convert_hf_checkpoint.py --checkpoint_dir checkpoints/$MODEL_REPO python quantize.py --checkpoint_path checkpoints/$MODEL_REPO/model.pth --mode int8 TOKENIZERS_PARALLELISM=false ENABLE_INTRA_NODE_COMM=1...