codebleu icon indicating copy to clipboard operation
codebleu copied to clipboard

Question about metric implementation

Open marcosmacedo opened this issue 6 months ago • 1 comments

Hi,

Thank you for sharing your CodeBLEU package. Could you please explain why is there an or 1 expression in the condition below in your implementation?

+ theta * (dataflow_match_score or 1)

https://github.com/k4black/codebleu/blob/4b8d576af9787bbc126db93df9d1e071898eaf85/codebleu/codebleu.py#L87

This is different from the XLCoST implementation.

https://github.com/reddy-lab-code-research/XLCoST/blob/ad46a7df51ea9e88f37a2f7e6edc5cbe4d13b2f2/code/translation/evaluator/CodeBLEU/calc_code_bleu.py#L76

During my testing I get a CodeBLEU score of 0.25 even if all the weights of the metric are zero. Is that the intended behavior?

Thank you

marcosmacedo avatar Aug 25 '24 14:08 marcosmacedo