conversationai-models icon indicating copy to clipboard operation
conversationai-models copied to clipboard

Converter to tf.record improvements

Open iislucas opened this issue 6 years ago • 1 comments

  • The current CSV to tf.record converter has hard-wired field selectors. These should be specified on the command line.
  • CSV is a bad format: there are multiple usually incompatible and badly supported 'standards'. If people use CSV, print out a warning, that CSVs are fragile, and the user should consider a more robust format like jsonlines, or json.
  • Support jsonlines, and json input, and when we do that, maybe rename script appropriately.
  • Consider: make our dataset class natively support and convert examples inline instead of requiring pre-processing.

iislucas avatar Jul 01 '18 01:07 iislucas

jsonlines support landed in https://github.com/conversationai/conversationai-models/pull/181

iislucas avatar Nov 22 '18 13:11 iislucas