← All projects

Word-Level vs. Character-Level N-gram Language Models

The most intuitive language model there is: count how often words (or characters) follow each other. Train one per class, and the model that finds a post more likely labels it. A fair comparison of unigram and bigram models at word and character level.

  • Language modelling
  • Natural Language Processing course
  • University of Tehran · 2021
  • Python · NLTK
N-gram language model banner
4models: uni/bigram × word/char
0.648best test accuracy: character bigram
0.98 → 0.57word bigram, train to test: memorisation
Laplaceadd-one smoothing for unseen n-grams

Try a class-conditional n-gram model

Two tiny language models, one per class

Runs in your browser

Illustration only: each class model is trained on a handful of toy sentences (shown below), not the project's dataset. Type a sentence and switch model types to see how the scores change.

Toy training sentences

Each token is coloured by which class model gives it higher probability (blue: polite, orange: rude); hover a token for both probabilities. The decision is the class with the higher total log-probability. Notice how the bigram model punishes word pairs it never saw, and how character level still scores misspelled or unseen words.

Results on the real data

Word-level models memorise. The word bigram is nearly perfect on training data (0.98 accuracy) and falls to 0.57 on test: most test word pairs never appeared in training, so smoothing gives them all the same tiny probability.

Character-level models generalise. Their train–test gap is small, and the character bigram is the best model on the test split (0.648 accuracy, 0.640 macro-F1). Characters are shared across words, so new words still get informative scores.

Word cloud of the training data