conversationai-models
conversationai-models copied to clipboard
Converter to tf.record improvements
- The current CSV to tf.record converter has hard-wired field selectors. These should be specified on the command line.
- CSV is a bad format: there are multiple usually incompatible and badly supported 'standards'. If people use CSV, print out a warning, that CSVs are fragile, and the user should consider a more robust format like jsonlines, or json.
- Support jsonlines, and json input, and when we do that, maybe rename script appropriately.
- Consider: make our dataset class natively support and convert examples inline instead of requiring pre-processing.
jsonlines support landed in https://github.com/conversationai/conversationai-models/pull/181