Saurabh Shah

I'll try to summarize learnings from [this project](https://docs.google.com/document/d/1cEO2bwus0cDQD28MwqL8UU2aEjI-s0eRUX373FYDKpE/edit?usp=sharing) below, hopefully I don't forget anything important. Here is the [WandB project](https://wandb.ai/ai2-llm/ReLoRA/) 1. **We were unable to reproduce the results in the...

Draft PR for this is here: https://github.com/allenai/LLM/pull/330 Right now it's about 40% slower when you turn on activation logging (which is a config flag). I hope this can be optimized...

Benchmarked different implementations of RoPE: 1. Base: what we have now, rotates in R² 2. TorchScript: Same as base but adds torch scripting to the `apply_rotary_pos_emb()` function 3. Complex: Instead...

This sounds to me like [Early Exit](https://arxiv.org/pdf/2306.02307.pdf) or [Adaptive Inference](https://arxiv.org/pdf/2306.02307.pdf). A brief lit review could be useful to anyone tackling this, especially for questions around how to combine (or not...

TODO: decide ourselves how to broadcast K and V tenors to match Q shape when using grouped-query-attention. To be revisited...

Ok, now that we merged in the PR which added async-by-default verifiers and configureable verifiers we're able to (finally) cleanly merge in the code verifier. There's code in `setup_ray_node.sh` which...

test run [here](https://beaker.allen.ai/orgs/ai2/workspaces/oe-adapt-code/work/01JWYXB3BK1MP1SVSTDGWNR1TX?taskId=01JWYXB3BR7AVNQGYPQSA92B5G&jobId=01JWYXB3GK19GY01CHX9J6WK9X) test wandb [here](https://wandb.ai/ai2-llm/open_instruct_internal/runs/5awi2cdo) reference wandb from previous (stable) code infra [here](https://wandb.ai/ai2-llm/open_instruct_internal/runs/48jil4fq)