← All projects

Tweet Sentiment with Hidden Markov Models

Instead of learning a decision boundary, learn how each kind of tweet is written: fit one hidden Markov model per sentiment, then ask which model makes a new tweet more likely.

  • Generative classification
  • Pattern Recognition course
  • University of Tehran · 2020
  • hmmlearn · TextBlob · NumPy
Hidden Markov model banner
2 HMMsone per sentiment class, 20 hidden states each
5,000training tweets (3,555 / 1,445 by class)
Baum–WelchEM training, 10 iterations per model
Forwardalgorithm also written from scratch in hmm.py

How the classifier works

A hidden Markov model explains a sequence of observed words with a sequence of unobserved states. Each state has its own vocabulary (the emission probabilities B), and states hand over to one another with transition probabilities A. Train one model on each class and you get two "writers" with different habits.

  1. TokeniseSplit each tweet into words with TextBlob and map every word to an index in a per-class vocabulary.
  2. Fit per classTrain a 20-state multinomial HMM on each class's tweets with Baum–Welch (EM).
  3. ScoreFor a new tweet, compute log P(words | HMMc) under both models with the forward algorithm.
  4. DecideAdd the log class prior and pick the larger: argmaxc log P(c) + log P(words | c).

The forward algorithm, step by step

Build a sequence and watch both models score it

Interactive

Toy parameters for illustration: 2 hidden states and 3 word types per model, with the class priors from the real training split (71% / 29%). Click words to append them; click a token to remove it.

Each column is a time step. A node's value is αt(j) = P(o1..t, st=j), computed as Σi αt-1(i)·Aij·Bj(ot). Edge thickness shows how much each predecessor contributes. P(sequence) is the sum of the last column.

What happened, honestly

The word-level HMM lost to a constant guess. On the 5,000 training tweets the classifier reached 44.4% accuracy, while always predicting the larger class would score 71.1%.

The model is not wrong in principle; the setup starved it. Each class had its own vocabulary, so most words seen in one class were unknown to the other; multinomial emissions over thousands of words with 20 states and only 10 EM iterations are badly under-trained; and a tweet's likelihood shrinks with every extra word, which punishes the model whose vocabulary is larger.

What I would change: a shared vocabulary with add-one smoothing, length-normalised log-likelihoods, fewer states, and dense word embeddings with Gaussian emissions instead of raw word indices.