text-feat-lib

text-feat-lib copied to clipboard

Text Features Library

Objective: Provide a comprehensive list of tokenizers, features, and general NLP things used for text analysis with examples. The initial focus on features used for twitter data and sentiment analysis.

Packages Used: nltk, spacy, pandas, scikit-learn

Notebooks

Features

(roughly in order of increasing complexity)

1. [Count of ALL CAPS words](notebooks/ALL CAPS.ipynb)

• [Count of Bible Verses](notebooks/Bible Verses.ipynb)
• [Word n-grams (Bag of Words)](notebooks/Word n-grams - Bag Of Words - BOW.ipynb)
• [Character n-grams](notebooks/Character n-grams.ipynb)
• [Hashtag Counts](notebooks/Hashtag Counts - Bag of Hashtags.ipynb)
• [Hashtag Bag of Words](notebooks/Hashtag BOW.ipynb)
• [Named Entity Counts](notebooks/Named Entities Count - Bag of Named Entities.ipynb)
• [Word n-grams - BOW - Sequence Split in Half](notebooks/Word n-grams - BOW - Sequence Split in Half.ipynb) ☨
• [Brown Word Cluster assignments](notebooks/TweetNLP - Brown Word Clusters.ipynb)
• [Bing Liu Lexicon Derived Features](notebooks/Bing Liu Lexicon Features.ipynb)
• [NRC Hashtag Sentiments (unigrams)](notebooks/Bing Liu Lexicon Features.ipynb)
• [NRC Hashtag Sentiments (bigrams)](notebooks/NRC Hashtag Sentiments - bigrams.ipynb)
• [NRC Emotion Lexicon Features](notebooks/NRC Emotion Lexicon Features.ipynb)
• [MaxDiff Twitter Sentiment Lexicon - unigrams and bigrams](notebooks/MaxDiff Twitter Sentiment Lexicon - unigrams and bigrams.ipynb) ☨
• [Sentiment 140 - unigrams](notebooks/Sentiment140 - unigrams.ipynb) ☨
• [Sentiment 140 - bigrams](notebooks/Sentiment140 - bigrams.ipynb) ☨

☨ - most recently updated

Notes on Features

• Each feature is demonstrated on the STS-Gold dataset (see Datasets below).
• Each feature is evaluated on accuracy using 5-fold cross validation with MultinomialNB (Multinomial Naive Bayes), BernoulliNB (Bernoulli Naive Bayes), and SVC (Support Vector Classifier) in comparison to selecting the most frequent class (DummyClassifier)
  • these are the most commonly used models I see for text classification tasks
• The lexicons used for some advanced features have links within the notebook as well as in the Lexicons table below.
  • They are not hosted in this repository to respect the wishes and licenses of their creators.

Tokenizers

• NLTK's casual tokenizer (TweetTokenizer)
  • Good for a tweet tokenization, takes hashtags, mentions, and emoji into account.
• NLTK's word_tokenize (default is TreebankWordTokenizer)
  • Good for simplicity's sake. The de-facto industry-standard Penn Treebank tokenization
• Christopher Potts Sentiment-Aware Tokenizer (HappyFunTokenizer)
  • Takes all tokens from NLTK's TweetTokenizer, and adds/changes:
    • HTML character resolution
    • HTML feature tokenization (ie. tags like <\b>)
    • Does not split contractions
  • Has negation approximating method for additional semantic features
  • Updated to work with python 3
• spaCy's default tokenizer (uses Penn Treebank)
  • Good for a faster tokenizing in comparison to NLTK's word_tokenize
• TweetNLP Tokenizer (python port)
  • Haven't used, I imagine it's similar to Sentiment-Aware and TweetTokenizer
  • Additional POS tagging features available on website

Datasets

Sentiment Classification

Source: Evaluation datasets for Twitter sentiment analysis: a survey and a new dataset, the STS-Gold

* transformations applied to convert various sentiment scores to sentiment labels

Lexicons

Lexicons are typically used for feature creation for sentiment classification tasks. Source: NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets

Literature / Papers

• NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets
  • Winners of the SemEval 2013/2014 Twitter Sentiment Analysis tasks describe their methodology
• NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets
  • Winners of SemEval 2013/2014 Twitter describe additional improvements to sentiment analysis due to negation contexts (handling ***n't, not, no, never, etc.)
• Harnessing Twitter ‘Big Data’ for Automatic Emotion Identification
  • Some interesting feature ideas like splitting a tweet in half and doing 2 BOWs
  • we also hypothesize that the words located towards the end of a tweet are more important than other words, because people usually summarize or highlight their points in the end. For example, “I hate it when stuff like that happens,.. ;/ thank god it worked out.<3 #thankful.”. Although “hate” appears in the first half of the tweet, the overall emotion is dominated by “thank” in the latter half. We encoded the position information into a feature by attaching a number (i.e, 1 or 2) to each n-gram to indicate whether it is in the first half or the second half of the tweet
• On Assessing the Sentiment of General Tweets
  • we include a new lexico-syntactic binary feature <w>-POS(w) where ‘w’ is a sentiment expressing word found in sentiment lexicons and POS(w) is its part-of-speech as observed in an input tweet. For such words, based on the dependency parse [5] of a tweet, we also introduce another lexico-syntactic binary feature <w>-<g/d>-<dtype>, where ‘dtype’ is the type of a dependency relation involving ‘w’ and ‘g/d’ is determined based on whether the relation has ‘w’ as a governor (g) or dependent