RedPajama-Data icon indicating copy to clipboard operation
RedPajama-Data copied to clipboard

Published 20 hours ago • verified publisher icon togethercomputer

fixed some errors in Makefile for lm preparation

Open feifeibear opened this issue 2 years ago • 2 comments

  1. install sentencepiece from github repo. I can not run the .zip version on my MacOS.
  2. make some necessary directories during make
  3. cache the wiki json.gz if has already been downloaded

feifeibear avatar May 08 '23 04:05 feifeibear

@mauriceweber could you review this PR?

feifeibear avatar May 08 '23 04:05 feifeibear

Hi @feifeibear , thanks a lot for your PR! I will look into it:)

mauriceweber avatar May 08 '23 16:05 mauriceweber