codebleu icon indicating copy to clipboard operation
codebleu copied to clipboard

wrong codeBLEU metric value

Open poojitharamachandra opened this issue 1 year ago • 0 comments

Hi,

I am comparing the code lines:

codeline1= 'a = !((f >> 4) & 0x01);' codeline2= 'a=!(((f >> 4) & 1U)!=0?true:false;)'

the metric is CodeBLEU score: {'codebleu': 0.4474481492943273, 'ngram_match_score': 0.21711852081087685, 'weighted_ngram_match_score': 0.21711852081087685, 'syntax_match_score': 0.5555555555555556, 'dataflow_match_score': 0.8}

When the change the code to codeline2='a=!(((f >> 4) & 1U)!=0?false:true;)' the metric is CodeBLEU score: {'codebleu': 0.4474481492943273, 'ngram_match_score': 0.21711852081087685, 'weighted_ngram_match_score': 0.21711852081087685, 'syntax_match_score': 0.5555555555555556, 'dataflow_match_score': 0.8}

I expected the dataflow match score to change, as the functionality is changed

poojitharamachandra avatar Feb 20 '24 08:02 poojitharamachandra