llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

Throw error when tokenization fails due to mistyped dataset entries.

Open milocress opened this issue 5 months ago • 0 comments

Catches tokenization failures on custom HF datasets with missing/extra/mistyped columns.

milocress avatar Sep 16 '24 22:09 milocress