About details for evaluation

[DZY-irene]

Thanks for your wonderful work! I have two questions about the evaluation:

1. if I want to directly utilize the pre-trained model for inference, do I need to do a prompt optimization before running cli_demo.py? I checked the pipe.encode_prompt function in cli_demo.py and it doesn't seem to do optimization.
2. When using VBench for evaluation, do you use a similar prompt optimization function for prompts from VBench?

[zRzRzRzRzRzRzR]

Our prompt optimization is implemented in convert_demo.py, not in cli_demo.py. You can refer to our Gradio demo for guidance on how to operate it. and for VBench yes the optimization is same as convert_demo.py(tested using GLM-4-0520)

[DZY-irene]

Thanks for your reply! I also want to confirm that you use the SAT version for evaluation right?

[zRzRzRzRzRzRzR]

yes

[DZY-irene]

yes

Thanks a lot!