BertScore giving different results each time

[p-H-7]

trafficstars

Hello @Tiiiger

Initially, i wrote my code such that it calls the score function multiple times, when applied to a dataframe, and it took a lot of time to compile. (Shown below)

def calculate_bertscore(row):
    source_text = row['Verbatim Translated']
    generated_summary = row['summary']

    summary_list = [generated_summary]
    source_list = [source_text]

    bertscore = score(summary_list, source_list, lang="en", model_type="bert-base-uncased", num_layers=4, device="cuda:0")

    f1_score = bertscore[0].item()

    return f1_score

Upon realizing my error by reading your reply, i modified my code to directly pass the columns as lists (as shown below).

summary_list = df_test_2['Verbatim Translated'].tolist()
source_list = df_test_2['summary'].tolist()

P, R, F1 = score(summary_list, source_list, lang="en", model_type="bert-base-uncased", num_layers=4, device="cuda:0")

df_test_2['F1-score'] = F1

df_test_2

The running time improved significantly but the results are comparatively bad. LIke the F1 scores for all the rows have dropped by approximately 0.1. Now, which of them would be the correct results, and I would like to know why there is such a drop in scores?

Thank you!