count-tokens-hf-datasets icon indicating copy to clipboard operation
count-tokens-hf-datasets copied to clipboard

Published 20 hours ago • verified publisher icon sayakpaul
→

Metadata

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Results 0 count-tokens-hf-datasets issues
Sort by recently updated
recently updated
newest added

← Metadata

22
Stars
1
Forks
Watchers

Owner

verified publisher icon sayakpaul

Metadata

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Back