open_lm icon indicating copy to clipboard operation
open_lm copied to clipboard

Published 20 hours ago • verified publisher icon mlfoundations

Update README on tokenization.

Open GeorgiosSmyrnis opened this issue 1 year ago • 0 comments

This PR does the following:

  • Consolidates the instructions on how to run tokenization in a single README.
  • Adds a sample script on how to run tokenization on a slurm cluster.

GeorgiosSmyrnis avatar Jan 15 '24 18:01 GeorgiosSmyrnis