llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

Add pre-tokenized Delta to MDS conversion script

Open mattyding opened this issue 11 months ago • 1 comments

This PR

Adds conversion script for pre-tokenized data in a Delta table.

Testing

MCLI IFT and CPT runs trained successfully.

mattyding avatar Nov 30 '24 06:11 mattyding