BBC-Esq

[SOLVED] Running Llama3 with Ctranslate2

4

I ran the new Llama3 sample script and it seems to be conversing with itself so I think there's a problem with how the prompt is being constructed...See below: ```...

Problem converting Phi3-instruct-128k; "su" rope scaling in Phi-3

1

Hello peeps, it's me again. The new converter works great with Phi3 but doesn't work with the 128k version located here: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct After much chagrin, I had a scintillating conversation...

BENCHmarking new flash attention!

10

Congrats on Flash Attention in the latest version, or to be precise, in having your storage limit increased on Pypi.org so you could upload the release that was weeks ago....

Support for Zephyr and other "StableLmForCausalLM" models?

2

Any plans to support conversion of ```StableLmForCausalLM" models? I've noticed that they're very good; for example the new Zephyr model here: https://huggingface.co/stabilityai/stablelm-zephyr-3b Amazing performance for a 3B model, much better...

Benchmarking common LLMs on ctranslate2, llama.cpp, and bitsandbytes

1

My initial testing comparing ct2 (using int8) and the ```bitsandbytes``` library at 4 and 8 bit...nicely done ctranslate2 people. Looking forward to testing GGUF in there as well.  Here's...

Hey dude, long time no see. How about some updated testing, which you promised but never did before? Here's a new repository that decimates "insanely". Have you heard of it?...

use av library instead baby!

2

Use the "av" library instead, which "just makes sense" since ```faster-library``` already uses it anyways. This removes the ```pyaudio``` dependency as well as that of ```scipy``` and ```numpy```. It also...

Put all tooltips, messages in window boxes, etc. in one or more dictionaries within ```constants.py``` instead of defining each one as they currently are. This will allow importing a single...

Tested and it has very good quality for the size: https://huggingface.co/openbmb/MiniCPM-V-2