CTranslate2
CTranslate2 copied to clipboard
OpenNMT
Fast inference engine for Transformer models
Congrats on Flash Attention in the latest version, or to be precise, in having your storage limit increased on Pypi.org so you could upload the release that was weeks ago....
A family of LLMs called [OpenELM](https://arxiv.org/abs/2404.14619) have recently been released. They range in size from 270M to 3B parameters: | **Model Size** | **ARC-c** | **ARC-e** | **BoolQ** | **HellaSwag**...
Thank you very much for your work. I'm using ctranslate2 accelerated https://huggingface.co/Maciel/T5Corrector-base-v2 reasoning and when using the cpu for inference the output is normal, but switching to using the GPU,...
Hello, everyone. I am trying to use ctranslate2 on android device. Hope any ctranslate2 expert to guide on how to use ctranslate2 on android. maybe we need to cross-compile ctranslate2...
Epsilon sampling is a compelling alternative/complement to top_p and top_k sampling and would make a good addition to CTranslate2: https://arxiv.org/abs/2305.09860
This has been a long time request. Recently resurfaced with #1534 . #1342 is also an example on how to perform classification but HF library is needed. Technically this is...
I am using the Orin-NX with CUDA version 11.4. The following error occurs during compilation: ``` (jarvis) aaeon@BOXER-8651AI:~/CTranslate2/build$ cmake .. -DWITH_MKL=OFF -DWITH_CUDA=ON -DWITH_CUDNN=ON -DOPENMP_RUNTIME=NONE -- The C compiler identification is...
Hello, how are you? I am building faster-whisper windows POC by ctranslate2. And i created ctranslate2::models::Whisper object whisperpool. and write the code such as: std::vector results; results = whisper_pool.generate(features, prompts,...
Any plans to support conversion of ```StableLmForCausalLM" models? I've noticed that they're very good; for example the new Zephyr model here: https://huggingface.co/stabilityai/stablelm-zephyr-3b Amazing performance for a 3B model, much better...
Hi, I'm new to ctranslate2, and I'm confused about the scores returned by generator.generate_batch() function. What's the coresponding meaning of the scores in the huggingface generate() function? For example, ```...
