Hynek Kydlíček

SSH with Proxy command not working on cursor

2

### Problem We are using scaleft to connect our clusters. The problem is that cursor doesn't seem to support it.... ``` Match exec "/usr/local/bin/sft resolve -q %h" ProxyCommand "/usr/local/bin/sft" proxycommand...

Klokan-qa task

1

### What is Klokan-qa - Dataset of mathematical questions in Czech for elementary and high-school children - ~850 questions - Dataset is not translated, which makes it the only publicly...

### This PR adds blocks for computing summary statistics - Added base block for creating the summary statistics - Added merging block for merging summary statistics computed in distributed manner....

## Problem #187 Introduced new tokenizer libraries, which will often need to download several files to work. This can however introduce a problems as the downloads are not interlocked. ##...

### Problem We currently use first x bytes sha1 for hashing, which is a waste of resources. - we don't need cryptographic guarantees - we only take first x bytes...

Translation literals

1

This PR provides translation literals for over 30 languages. ``` ara - Arabic (true/false/neither, or) eus - Basque ben - Bengali (only question_word is set) zho - Chinese (true/false/neither, or,...

## Issue encountered - If all choices for loglikehood task are exactly one token, one can only run single pass to compute their logprobs. This is the case of MCF...

[FT] Remove obsolete config properties (frozen, output_regex)

1

## Request Removed obselete config properties: - frozen - output_regex For more context see: https://github.com/huggingface/lighteval/pull/289 ``` the output_regex is legacy from the harness, we needed it for bigbench tasks but...

## Issue encountered When one wants to launch all MMLU tasks from cli, he needs to specify all subsets to get the correct average. This is very user unfriendly and...

## Describe the bug Running nanotron with batch_size = 0 causes nanotron to crash during batch detection. ``` (lighteval-main) hynek_kydlicek@ip-26-0-162-233:/fsx/hynek_kydlicek/projects/lighteval-main-branch$ torchrun --standalone --nnodes=1 --nproc-per-node=1 src/lighteval/__main__.py nanotron --checkpoint_config_path ./nanotron/checkpoints/0/config.yaml --lighteval_config_path examples/nanotron/lighteval_config_override_template.yaml...