the-pile icon indicating copy to clipboard operation
the-pile copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

(Natural) Languages in The PILE

Open suzyahyah opened this issue 2 years ago • 1 comments

Hi,

Has there been any Language ID of the sentences in PILE, and also quantifying their proportions? We can get an idea from Europarl, but it is less clear with Common crawl in the mix.

I have not seen this in any of the official documentation or the paper. If I missed something please let me know.

Thanks!

suzyahyah avatar Apr 14 '22 23:04 suzyahyah