about customize

[w-JiqQian]

Can I customize the indicators related to my data set in this file? #!/bin/bash

CUDA_VISIBLE_DEVICES=0
deepspeed examples/evaluate.py
--answer_type medmcqa
--model_name_or_path /root/autodl-tmp/model/BEELE/
--dataset_path /root/autodl-tmp/LMFlow/data/alpaca/test
--deepspeed examples/ds_config.json
--metric ppl

[research4pan]

Thanks for your interest in LMFlow! If you would like to change dataset path, you may set --dataset_path to {path-to-your-dataset}/*.json, where *.json follows "text_only" typed format.

If you mean changing metrics, currently we support "ppl" (perplexity) and "accuracy" metric for evaluation, where perplexity requires a "text_only" typed dataset and accuracy requires a "text2text" typed dataset. This may require further modification of the code in src/lmflow/pipeline/evaluator.py. If you need modification of this part, we would be happy to provide more information.

Thanks 😄

[w-JiqQian]

In addition,

1. will the type of “--answer_type” affect model training, and how to define my type?
2. How to load and use the model trained by Lora?

I would be very grateful if you could answer these questions

[w-JiqQian]

Thanks for your interest in LMFlow! If you would like to change dataset path, you may set --dataset_path to {path-to-your-dataset}/*.json, where *.json follows "text_only" typed format.

If you mean changing metrics, currently we support "ppl" (perplexity) and "accuracy" metric for evaluation, where perplexity requires a "text_only" typed dataset and accuracy requires a "text2text" typed dataset. This may require further modification of the code in src/lmflow/pipeline/evaluator.py. If you need modification of this part, we would be happy to provide more information.

Thanks 😄

Yes, I want to add the indicators I want by modifying this script(https://github.com/OptimalScale/LMFlow/blob/main/src/lmflow/pipeline/evaluator.py)

[research4pan]

Then you may add an extra metric starting from this line. You can add your own metrics by referring to this pull request, which added a new metric called negative log likelihood.

Hope that answers your question. Thanks 😄

[shizhediao]

This issue has been marked as stale because it has not had recent activity. If you think this still needs to be addressed please feel free to reopen this issue. Thanks