minbpe
minbpe copied to clipboard
add error handling for load method
Add error handling to the load
method in base.py
, specifically handling cases where the model file format might be incorrect or corrupted. Modifications include:
- Checking the model file version and raising a ValueError if it's unsupported.
- Verifying the format of lines when reading special tokens and merges, and raising ValueError for invalid lines.
- Using a more specific RuntimeError for unexpected errors during file processing.