sglucas

Results 7 comments of sglucas

Results on WSC and WIC datasets cannot be reproduced on OPT-13B with MeZO

Hi, May I ask for the seed of WSC and WIC. I find the default seed is 0 and I cannot reproduce the result of WIC 61.1.

Is it possible to release a code based on jax?

@lucidrains Hi, I'm supported by TRC program and I can use TPUs to do some research. Thank you very much for your contribution.

[Feature] I try to use instruction tuning to fine-tune llama model and get a new alpaca model. How to use compass to evaluate the local alpaca model on MMLU and other datasets

Hi, huggingface format

Can not reproduce results on the table

Hi, May I ask the update about this issue.

Suggestions on making Qwen3-VL evaluation prompts/scripts public，benchmark Failed to reproduce

I have the same question. It could be better if you can provide more details about reproducing the results.

A bug in qwen3vl

Is there any evaluation code about Qwen3-VL? I try to search on GitHub, but it's difficult to find running code that we can directly use. The official evaluation code from...

About DQN replay dataset

Hi, @agarwl Thank you very much for your reply. I'll read these papers. Best