Kemal Cihan Baştak comments

Repositories
Issues
Comments

Results 1 comments of


                                            Kemal Cihan Baştak

What purpose cutoff.csv used in the cc_net pipeline?

I have calculated for 2023-50 CC dump and used 'perplexity' score on that data. `percentiles = {f"%{i}": np.percentile(all_pp_values, i) for i in range(1, 101)}` Got closer values with the existing...