Luca Antiga

Yes it would be great It would be cool to use Textual for the UI https://www.textualize.io/#textual

Hey, lit-GPT supports it https://github.com/Lightning-AI/lit-gpt/blob/main/tutorials/download_mistral.md with capped context length (which should be enough for most practical uses right now)

Hey @DuarteMRAlves I don't disagree. It should be fairly doable to take the current conversion script and rearrange the state dict. Help welcome :-)

Hey @Arnav0400, it does work at least for HellaSwag and PiQA, which are the tests I focused on a few weeks ago https://github.com/EleutherAI/lm-evaluation-harness/issues/539 I'm now also adding lit-parrot support as...

I gave it a shot and the benchmarks seem to run on lit-llama. I'll publish an howto, but for the time being, this is the end-to-end rundown (I'm downloading OpenLLaMA...

yes and no :-| the issue is that to go on the leaderboard HF requires the models to be on the HF hub (i.e. to be integrated within HF transformers)...

So what will unblock that specifically is a conversion script of checkpoints from lit-* to equivalent HF models, so you can upload on the hub. We’ll try to schedule this...

It's quite possible because I'm only supporting `batch_size` of 1 atm (I was only interested in the numbers at the time so that was enough for me)

We removed that to make the code simpler, with the idea to add a separate script in case it was of interest (it looks like there is interest :-) )....

My understanding is you introduce a token that wasn't there during pre-training, that takes the specific meaning of ending both instruction and response. So the prompt will be in the...