minimind
minimind copied to clipboard
jingyaogong
Eval Ⅲ Objective Benchmark 部分报错，请问可以提供一个示范性的配置文件吗？
lm_eval --model hf --model_args pretrained='./out/',device=cuda,dtype=auto --tasks ceval* --batch_size 8 --trust_remote_code
2025-02-21:18:13:52,363 INFO [lm_eval.tasks:460] The tag 'kobest' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2025-02-21:18:13:52,365 INFO [lm_eval.tasks:460] The tag 'kobest' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2025-02-21:18:13:52,367 INFO [lm_eval.tasks:460] The tag 'kobest' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2025-02-21:18:13:52,369 INFO [lm_eval.tasks:460] The tag 'kobest' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2025-02-21:18:13:52,371 INFO [lm_eval.tasks:460] The tag 'kobest' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2025-02-21:18:13:56,686 INFO [lm_eval.main:368] Passed --trust_remote_code, setting environment variable HF_DATASETS_TRUST_REMOTE_CODE=true
0
Selected Tasks: ['ceval-valid', 'ceval-valid_accountant', 'ceval-valid_advanced_mathematics', 'ceval-valid_art_studies', 'ceval-valid_basic_medicine', 'ceval-valid_business_administration', 'ceval-valid_chinese_language_and_literature', 'ceval-valid_civil_servant', 'ceval-valid_clinical_medicine', 'ceval-valid_college_chemistry', 'ceval-valid_college_economics', 'ceval-valid_college_physics', 'ceval-valid_college_programming', 'ceval-valid_computer_architecture', 'ceval-valid_computer_network', 'ceval-valid_discrete_mathematics', 'ceval-valid_education_science', 'ceval-valid_electrical_engineer', 'ceval-valid_environmental_impact_assessment_engineer', 'ceval-valid_fire_engineer', 'ceval-valid_high_school_biology', 'ceval-valid_high_school_chemistry', 'ceval-valid_high_school_chinese', 'ceval-valid_high_school_geography', 'ceval-valid_high_school_history', 'ceval-valid_high_school_mathematics', 'ceval-valid_high_school_physics', 'ceval-valid_high_school_politics', 'ceval-valid_ideological_and_moral_cultivation', 'ceval-valid_law', 'ceval-valid_legal_professional', 'ceval-valid_logic', 'ceval-valid_mao_zedong_thought', 'ceval-valid_marxism', 'ceval-valid_metrology_engineer', 'ceval-valid_middle_school_biology', 'ceval-valid_middle_school_chemistry', 'ceval-valid_middle_school_geography', 'ceval-valid_middle_school_history', 'ceval-valid_middle_school_mathematics', 'ceval-valid_middle_school_physics', 'ceval-valid_middle_school_politics', 'ceval-valid_modern_chinese_history', 'ceval-valid_operating_system', 'ceval-valid_physician', 'ceval-valid_plant_protection', 'ceval-valid_probability_and_statistics', 'ceval-valid_professional_tour_guide', 'ceval-valid_sports_science', 'ceval-valid_tax_accountant', 'ceval-valid_teacher_qualification', 'ceval-valid_urban_and_rural_planner', 'ceval-valid_veterinary_medicine']
2025-02-21:18:13:56,687 INFO [lm_eval.evaluator:169] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-02-21:18:13:56,687 INFO [lm_eval.evaluator:206] Initializing hf model, with arguments: {'pretrained': './out/', 'device': 'cuda', 'dtype': 'auto', 'trust_remote_code': True}
2025-02-21:18:13:56,744 WARNING [accelerate.utils.other:335] Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2025-02-21:18:13:56,744 INFO [lm_eval.models.huggingface:136] Using device 'cuda'
Traceback (most recent call last):
File "/home/dev/.conda/envs/llm/bin/lm_eval", line 8, in model_type key in its config.json, or contain one of the following strings in its name: albert, align, altclip, aria, aria_text, audio-spectrogram-transformer, autoformer, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, colpali, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, deta, detr, diffllama, dinat, dinov2, dinov2_with_registers, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, emu3, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, git, glm, glpn, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, graphormer, grounding-dino, groupvit, hiera, hubert, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rwkv, sam, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, siglip, siglip_vision_model, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, textnet, time_series_transformer, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zamba, zoedepth
