James

Constrained beam search allows the user to exert control over the output of text generation based on forcing certain terms (like phrase table entries). Currently beam search limits the user...

VRAM usage not constant in v2.0.0rc1

3

Using the same (Transformer Big) model parameters in 1.2.0 and 2.0.0rc1, my 3090 24GB will run out of memory at random times during the training process (uses from 75% to...

Implement Constrained Beam Search (Disjunctive Positive Constraint Decoding)

1

### Feature description Constrained beam search allows the user to exert control over the output of text generation based on forcing certain terms (like phrase table entries). Currently beam search...

Plans to implement HF's int8 inference?

1

Would be great if someone could look into implementing this particular version of int8 for serving LLMs. https://huggingface.co/blog/hf-bitsandbytes-integration

Is there a way to export all the fields, with each Text # being a new line? i.e., there will be a `bccent_1.txt` with contents ``` “So peaceful here. Yet...

Support Speculative Decoding

5

This could be used for LLMs and hopefully for encoder-decoder models like using the smaller NLLB coupled with the bigger NLLB models

Splitting LLM layers across multiple GPUs

3

As CTranslate2 now supports quantized 8-bit LLMs like OPT, are there any plans to include model parallelism to split a model layers across multiple GPUs or GPU+CPU to meet the...

### How does pos_list work for Force Atlas 2? I'm trying to specify the initial positions of the nodes for Force Atlas 2 based on their community. I've already run...

It seems like the MsgViewer work on reading `message.win32.msgbin` wasn't continued?