tf*idf: Why don't you normalize to maximum term count for document?

[jpmckinney]

Treat (and the tf-idf and similarity gems) all normalize tf to the number of terms in the document: https://github.com/louismullie/treat/blob/master/lib/treat/workers/extractors/tf_idf/native.rb#L78

We normalize so that (1) long and short documents have comparable tf weights and (2) documents with large vocabularies and those with small vocabularies have comparable tf weights.

Normalizing to the number of terms in a document only fixes (1). Normalizing to the maximum term count (which is all I've ever seen in the literature) fixes both (1) and (2).

[louismullie]

It should be included in the next release of the gem.

[jpmckinney]

I wrote my own gem in the meantime: https://github.com/opennorth/tf-idf-similarity Perhaps you'd like to re-use it?

[louismullie]

Sweet. I will! My philosophy is really to keep as much logic code as possible separate from the Treat core, so this is definitely the kind of contribution I welcome. My naïve implementation was horribly inefficient anyway.

Thanks, Louis

[louismullie]

I am interested in using your library to replace the current tf_idf workers, but my requirements would be as follows:

1 - We need to be able to input documents that are already tokenized. 2 - We need to be able to easily access tf*idf scores for a given word in a given document.

From what I understand, the gem does not provide a public interface to perform either of these tasks.

Are you interested in coding up the public functions so I can use it directly? Or should I add it to my (long) to-do list for Treat?

[jpmckinney]

1. I assume the tokenized documents would be arrays? You can now do that by passing :tokens argument when initializing a document, eg. Document.new("Lorem ipsum", :tokens => ["Lorem", "ipsum"])
2. If you have a collections of documents, you can now get the tf*idf for a particular document and term with collection.tfidf(document, term). Previously you'd have to multiply the idf and tf yourself, eg:

collection.idf(term) * collection.tf(document, term)