Ian Magnusson
Ian Magnusson
### System Info - `transformers` version: 4.20.1 - Platform: Linux-5.4.0-89-generic-x86_64-with-glibc2.31 - Python version: 3.9.12 - Huggingface_hub version: 0.8.1 - PyTorch version (GPU?): 1.12.0+cu113 (False) - Tensorflow version (GPU?): 2.9.1 (False)...
The llm-eval team is building out scripts to convert evaluation data to a common format for the purposes of deduplication. This format is line separated json for each eval example...
This PR makes use of new features in Catwalk's perplexity evaluations in https://github.com/allenai/catwalk/pull/155 that report avg logits for tokens.
A few small changes for clarity
### 🚀 The feature, motivation and pitch Yesterday we spoke about where responsibility for data order lives between the llm-model and llm-data workstreams. I thought it might be good to...
Fixes # Changes proposed in this pull request: - ## Before submitting - [ ] I've read and followed all steps in the [Making a pull request](https://github.com/allenai/ai2-llm-eval/blob/main/.github/CONTRIBUTING.md#making-a-pull-request) section of the...
Todo: - [x] make Nvidia image based docker file that allows anaconda set up afterwards - [ ] write up anaconda reqs for Wav2Lip and evaluation code into README.md as...
Making a separate issue to track our progress on experimenting with Perceptual Hashing as a new metric. @arunasank suggested this idea and has begun some initial experiments with this on...
## What is here Fixes support for distributed training with data parallelism. Previously torch metrics would attempt to synchronize across processes during validation call back and would cause a crash....
## What's Here Moves a more generalized IA3 adaptor implementation to Tango ([PR pending](https://github.com/allenai/tango/pull/403)) and provides an example script for how to use it in Catwalk. ## Results on piqa...