FastChat
FastChat copied to clipboard
Category Hard Data Labeling Script
trafficstars
Why are these changes needed?
Related issue number (if applicable)
Checks
- [x] I've run
format.shto lint the changes in this PR. - [x] I've included any doc changes needed.
- [x] I've made sure the relevant tests are passing (if applicable).
At the moment, I am taking in cache file and output file as jsonl file unless there is a more effective way at writing to json file row by row.
Also make sure to set ulimit -n 16384 or other limit accordingly when running larger parallel.