obsidian-tagcloud icon indicating copy to clipboard operation
obsidian-tagcloud copied to clipboard

Ability to purge low count keywords to reduce data.json size

Open jwhco opened this issue 2 years ago • 2 comments

The data.json file is getting larger than 100 MB limit on GitHub. Looking into the file I see single word keyword table. Keywords like "He", "a", and "is" don't add value to tag clouds. An option to omit keyword counts under a certain number and/or words under a certain number of characters would reduce file size. This kind of enhancement will lead to a smaller file size plus help user tune tag cloud.

jwhco avatar Dec 05 '22 20:12 jwhco

It would also be useful to have an upper limit on keyword counts. Some keywords with thousands of counts are low value. At a minimum short words tend to have least value, especially transition words and verbs.

jwhco avatar Dec 05 '22 20:12 jwhco

The big problem is maintaining both withStopwords and withoutStopwords in the data.json for every document in a repository. The trim of word length needs to happen before that point. This will reduce the file size. Plus, if there is one instance of something, I'd rather purge that then note it.

jwhco avatar Dec 05 '22 20:12 jwhco