any one has evaluate the performance of programming?

[friendmine]

any one has evaluate the performance of programming? and how to fine tuning a programming language, such as Java, C or others?

[friendmine]

maybe it's a rare question...

[zhisbug]

@friendmine

• In terms of evaluation -- you can run vicuna on the human-eval benchmark. Since we do not specialize it to coding, I believe there exist many better alternative models better than vicuna on coding.
• In terms of how to tune it for another programming language, I guess you need to curate a fine-tuning dataset in java and C and use that dataset for fine-tuning. You can try to create your own dataset and use the scripts in fastchat to perform the fine-tuning.

I hope I have answered your questions.

[friendmine]

thanks for you answer. I'll try to make a dataset.