COMET
COMET copied to clipboard
Unbabel
A Neural Framework for MT Evaluation
## 🚀 Feature HF transformers implements 8 bit and 4 bit quantization. It would be nice if that feature can be leveraged for the xlm-r-xxl machine translation eval model. ###...
Using comet-mbr for Multi-Model Translation Ranking: Questions About Input Format and GPU Disabling
I'm attempting to utilize the following command: `comet-mbr -s [SOURCE].txt -t [MT_SAMPLES].txt --num_sample [X] -o [OUTPUT_FILE].txt` for the purpose of ranking multi-model translations. However, I have a couple of questions:...
I noticed that `comet-train` (after encoder finetuning) has speed of ~12it/s at e.g. 30% which drops to ~7it/s at 60% and to ~6it/s at 90% of the epoch. 1. Is...
## 🐛 Bug Currently, the library does not allow sentences that contain newline characters (i.e. '\n') but rather will split them into subsentences to compute scores. This is due to...
## 🐛 Bug wmt20-comet-qe-da do not work under huggingface guides I followed https://huggingface.co/Unbabel/wmt20-comet-qe-da, ``` model_path = download_model("wmt20-comet-qe-da") # where I removed Unbabel according to https://github.com/Unbabel/COMET/issues/125 since I use comet 1.1.3...
## 🐛 Bug I tried using [Huggingface evaluate](https://huggingface.co/docs/evaluate/index) but it is not working with the newest `Unbabel/wmt22-comet-da` as part of unbabel-comet `2.0.0` release. I'm not sure if this is an...
## 🚀 Feature PyTorch 1.12 came with a speed bump in Transformer encoder inference speeds through [BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/). Because of Unbabel's efforts to improve usability and efficiency (~COMETINHO), such improvements might...
## 🚀 Feature being able to specify the batch size in terms of tokens could allow larger batch sizes. ### Motivation When training on data with various sentence lengths, increasing...
What is the reason for this ? here: https://github.com/Unbabel/COMET/blob/master/comet/encoders/bert.py#L67 thanks.
## ❓ Questions and Help ### Before asking: 1. Search for similar [issues](https://github.com/Unbabel/COMET/issues). 3. Search the [docs](https://unbabel.github.io/COMET/html/index.html). #### What is your question? I can predict scores with only cpu successfully....
