COMET

COMET copied to clipboard

Hide device info prints in Python

7

## 🐛 Bug / feature(?) Whenever running COMET from the `huggingface-evaluate` library, I get the following output: ``` GPU available: True, used: True TPU available: False, using: 0 TPU cores...

zouharvi

## 🐛 Bug Encoding of io operations is done inconsistently. Scoring is performed expecting `utf-8`, but MBR decoding is not (see lines [188](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/mbr.py#L188) and [191](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/mbr.py#L191)), and compare only sometimes (see...

ioverho

Add a check if language is supported by model

15

Hello, In my ongoing evaluation of metrics, I have found out, that COMET (especially source based) is very unpredictable when it evaluates a language that is not supported by XLM...

kocmitom

Multi-system Evaluation

3

## 🚀 Feature It will be great if COMET makes a multi-system evaluation both with Paired Bootstrap Resampling and Paired Approximate Randomization for paired significance tests. ### Motivation Paired Approximate...

andmek

Problems with MULTI-GPU scoring

1

## 🐛 Bug Hi, there! I found that there were some problems with MULTI-GPU usage with COMET. When using 4 GPUs the segments seem to be getting unordered, making it...

c-soler-u

Read from STDIN

6

## 🚀 Feature It would be really nice if COMET could read input from STDIN, e.g., ```bash # three fields triggers comet-ref $ paste source.txt hyps.txt ref.txt | comet [args]...

mjpost

## 🐛 Bug Passing `--to_json` as `True` or `true` doesn't output any JSON file. Whereas passing any other value creates a file with that name and it has the JSON...

me-manikanta

## 🚀 Feature When running `comet-compare` show the actual COMET score for each system. Is this possible? ### Motivation I usually run the compare to know if a system is...

ZJaume

[QUESTION] Splitting big models over multiple GPUs

6

When specifying the number of GPUs during inference, is it only for parallelism or is the model loaded piece-wise over multiple GPUs, if it's bigger than individual GPUs? For example...

zouharvi

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2. Release notes Sourced from aiohttp's releases. 3.9.2 Bug fixes Fixed server-side websocket connection leak. Related issues and pull requests on GitHub: #7978. Fixed web.FileResponse...

dependabot[bot]