minbpe icon indicating copy to clipboard operation
minbpe copied to clipboard

add error handling for load method

Open halannhile opened this issue 11 months ago • 0 comments

Add error handling to the load method in base.py, specifically handling cases where the model file format might be incorrect or corrupted. Modifications include:

  1. Checking the model file version and raising a ValueError if it's unsupported.
  2. Verifying the format of lines when reading special tokens and merges, and raising ValueError for invalid lines.
  3. Using a more specific RuntimeError for unexpected errors during file processing.

halannhile avatar Feb 29 '24 22:02 halannhile