tensorfx
tensorfx copied to clipboard
Implement a schema field of type tokens
This is for representing tokenized set of values (eg. comma separated strings or numbers). The ideas is as part of data preparation, one can convert text into a set of tokens (after doing word breaking, stop-word removal, stemming etc.)