Yeskendir Koishekenov comments

Results 5 comments of


                                            Yeskendir Koishekenov

Generating config.json in olmo_data/oe_eval_tasks for new evaluation tasks.

Thanks for pointing out how to generate the request files. Regarding the second step, I don't see **references for evaluating generative tasks such as GSM8K** at https://github.com/allenai/OLMo/blob/main/olmo/eval/downstream.py#L1611. It seems to...

Generating config.json in olmo_data/oe_eval_tasks for new evaluation tasks.

Hello! I am trying to evaluate the models I have trained or fine-tuned using the code in this repository, rather than models from Hugging Face. It seems that OLMES works...

Generating config.json in olmo_data/oe_eval_tasks for new evaluation tasks.

Hi! Thanks for your reply. However, I am trying to evaluate the model on **generative tasks (e.g. GSM8K)** during training, e.g. after each eval_steps during training. I assume that during...

Generating config.json in olmo_data/oe_eval_tasks for new evaluation tasks.

Thanks for the clarification! In my case, I modified the OLMo() model architecture and trained it using the OLMo pipeline (data, code, and hyperparameters). As I understand, olmes currently supports...

Generating config.json in olmo_data/oe_eval_tasks for new evaluation tasks.

Hi! I tried running the command you provided with the following arguments: ``` olmes --task arc_challenge::olmes \ --output-dir eval_results/arc_challenge/olmo-1b-local-test \ --model "/home/local/folder/checkpoints/OLMo-2-0425-1B/step1907359" ``` I downloaded the checkpoint from the url...