Format

[arjunmenon]

Hey Can you detail further the text format to use. From the sample it looks like a tokenized lowercased and lemmatized text without stopwords? Is that all?

Doing it on my sample text, gives a ArgumentError: invalid slice size Text is

quantum mechanics is the part of physics that tells how the things that make up atoms work qm also tells how electromagnetic waves like light work it is also called quantum physics or quantum theory qm is a mathematical framework rules written in maths for much of modern physics and chemistry quantum mechanics helps make sense of the smallest things in nature like protons neutrons and electrons complex mathematics is used to study subatomic particles and electromagnetic waves because they act in very strange ways quantum mechanics is important to physics and chemistry

[watzon]

@arjunmenon this may be a little late, but the case doesn't really matter, nor does the format in general. Just try to make sure all of the text is relevant, i.e. don't include HTML, links, headings, etc.