candle

candle copied to clipboard

This issue aims at keeping track of the models that would be interesting to get added to candle. Feel free to make a comment to mention a new model, or vote for a model already in the list.

• musicgen.
• nougat, donut, see #1070.
• LCM Dreamshaper v7, fast diffusion model.

Added recently:

• JinaBert, embeddings model with an apache 2 license and a larger context (+ using alibi biases rather than rope embeddings).
• Marian-MT, a neural machine translation model.
• Yi-6b / Yi-34b, bilingual (English/Chinese) LLM.

Decent images with 4 steps of inference

• Stable Diffusion LCM

Apache licensed and has a fairly large community. Perhaps a minimal port as an example.

• Tortoise TTS

StyleTTS2 MIT licensed (Possibly better and faster than tortoise tts)

• Demo
• Github

The new JinaBert Embeddings is small and has Apache license

The new JinaBert Embeddings is small and has Apache license

This looks like one I'd love to help migrate; @LaurentMazare I can create an issue and get started on this?

The new JinaBert Embeddings is small and has Apache license

This looks like one I'd love to help migrate; @LaurentMazare I can create an issue and get started on this?

Ah sorry actually I'm mostly done with it, see #1187 (though I still have to line up the embedding values properly but all the rest is in place) - I had a couple hours in the train this afternoon :)

The new JinaBert Embeddings is small and has Apache license

This looks like one I'd love to help migrate; @LaurentMazare I can create an issue and get started on this?

Ah sorry actually I'm mostly done with it, see #1187 (though I still have to line up the embedding values properly but all the rest is in place) - I had a couple hours in the train this afternoon :)

Late to the party 😅, I'd keep an eye out for this list then.

@radames the jina-bert bits have been merged and I checked on some examples that the generated embeddings line up properly with the python version so should be all good. I will just polish the example a bit to properly download the tokenizer and weight files from the hub if needed. @ToluClassics and beside the list, if you have some models that you would feel interested by, that's certainly a great way to get started.

can support https://github.com/infinitered/nsfwjs or https://github.com/bhky/opennsfw2 for nsfw detection?

Would it be possible to show how to use a marian translation model in candle ?

There is already an example in :

• https://github.com/guillaume-be/rust-bert/blob/main/examples/translation_marian.rs
• https://github.com/guillaume-be/rust-bert/blob/dc99a30204ffcef98aee3f697ac90513ad67773d/src/models/marian/marian_model.rs#L4

The modeling_marian.py modeling file is already available in transformers :

• https://github.com/huggingface/transformers/blob/main/src/transformers/models/marian/modeling_marian.py

Marian translation models being lighter than their counterparts they are well-suited for serverless application. Candle being lighter than rust-bert and relying less on tch-rs I expect this would lighten and ease the whole build process.

@flutter-painter the marian-mt model should now be available in candle, we have an example that uses it to translate from french to english which you can find here.

Thanks @LaurentMazare I just tested and it works. You are blazingly fast !

@flutter-painter the marian-mt model should now be available in candle, we have an example that uses it to translate from french to english which you can find here.

thanks for amaing works, but could you please tell me how to get tokenzier-marian-{lang}.json ? i try to get from python code

tokenizer = MarianTokenizer.from_pretrained(model_name)
tokenizer.save_pretrained("./out")

but it does not work. I noticed that there is that file "convert_slow_tokenizer.py", do I need to use the function of this file? how to use? thanks very much!

thanks for amaing works, but could you please tell me how to get tokenzier-marian-{lang}.json ? i try to get from python code

tokenizer = MarianTokenizer.from_pretrained(model_name)
tokenizer.save_pretrained("./out")

but it does not work. I noticed that there is that file "convert_slow_tokenizer.py", do I need to use the function of this file? how to use? thanks very much!

Ah that part is indeed very hacky at the moment. You have to download this convert_slow_tokenizer.py script that you discovered and from the same directory running the following python code shoud produce the two tokenizer files.

from convert_slow_tokenizer import MarianConverter
from transformers import AutoTokenizer


tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en", use_fast=False)
fast_tokenizer = MarianConverter(tokenizer, index=0).converted()
fast_tokenizer.save(f"tokenizer-marian-base-fr.json")
fast_tokenizer = MarianConverter(tokenizer, index=1).converted()
fast_tokenizer.save(f"tokenizer-marian-base-en.json")

ChatGLM3 huggingface : https://huggingface.co/THUDM/chatglm3-6b

ChatGLM3 huggingface : https://huggingface.co/THUDM/chatglm3-6b

That sounds like a great model to have, happy to prioritize it. Do you know if the tokenizer config can be easily converted to a working tokenizer.json or maybe there is already such a config somewhere? (similar to what was done for marian but hopefully less flaky)

How are we doing on TTS? Would be nice to have e.g. Bark.

Also, +1 on nougat

anather embeddings model https://huggingface.co/moka-ai/m3e-large

anather embeddings model https://huggingface.co/moka-ai/m3e-large

I think this one may just work out of the box as it uses a standard bert model which already has been added. You could try it out with the following:

cargo run --example bert -- --model-id moka-ai/m3e-large --revision refs/pr/5

anather embeddings model https://huggingface.co/moka-ai/m3e-large

I think this one may just work out of the box as it uses a standard bert model which already has been added. You could try it out with the following:

cargo run --example bert -- --model-id moka-ai/m3e-large --revision refs/pr/5

it works , thank you!

@LaurentMazare another LLM https://github.com/01-ai/Yi , they coverted the tokeninzer.json , i tested with Tokenizers , https://github.com/01-ai/Yi/issues/24#issuecomment-1801680600 , can candle support this Model?

@LaurentMazare another LLM https://github.com/01-ai/Yi , they coverted the tokeninzer.json , i tested with Tokenizers , 01-ai/Yi#24 (comment) , can candle support this Model?

Do you know if this is the same tokenizer as for ChatGLM3? If that's the case I would prefer pushing on this one first as I have a PR that should be mostly done with the implementation and only requires lining up the logits once we have a proper tokenizer config.

(edit: I misremembered the PR, it's not mostly complete and requires a bit of work implementing the forward passes but this should be pretty quick to do once we have a tokenizer to test out)

@LaurentMazare another LLM https://github.com/01-ai/Yi , they coverted the tokeninzer.json , i tested with Tokenizers , 01-ai/Yi#24 (comment) , can candle support this Model?

Just merged support for the yi-6b and yi-34b variants in #1320 , I haven't tested them much though as my current computer is very slow even on the 6b - not sure how much of that is expected. It would certainly be great if you can give these a spin and let me know how it goes.

@LaurentMazare another LLM https://github.com/01-ai/Yi , they coverted the tokeninzer.json , i tested with Tokenizers , 01-ai/Yi#24 (comment) , can candle support this Model?

Just merged support for the yi-6b and yi-34b variants in #1320 , I haven't tested them much though as my current computer is very slow even on the 6b - not sure how much of that is expected. It would certainly be great if you can give these a spin and let me know how it goes.

thank you very much, i'll do some testing and i'll follow your pr , try to convert other LLMs

Just for curiosity, is there any plan for supporting more embedding models such as deberta or BAAI/bge-large-en-v1.5?

Hey :wave:, is someone working on porting LCM to Candle ?

Just for curiosity, is there any plan for supporting more embedding models such as deberta or BAAI/bge-large-en-v1.5?

As this seems to just be a bert variant, this could work directly with the current bert model provided by candle.

cargo run --example bert --release -- --model-id BAAI/bge-large-en-v1.5 --revision refs/pr/5

Hey 👋, is someone working on porting LCM to Candle ?

Could you provide some links/details on what LCM is?

Hey 👋, is someone working on porting LCM to Candle ?

Could you provide some links/details on what LCM is?

Yep, I opened an dedicated issue here

EfficientSAM: https://github.com/xetdata/EfficientSAM

Re text-to-speech, we've just added an early version of metavoice, you can try it out via this example.

Would anyone has interest for running moondream2 if it was added to candle? Looks like a small and efficient model that wouldn't be too hard to add.