← All projects

Hate Speech Detection with BERT and XLNet

Platforms prohibit hateful posts, and on Twitter a single anonymous line is enough to cause harm. Two pretrained transformers, a masked language model and a permutation language model, fine-tuned as hate-speech classifiers and compared head to head.

  • Transformers · fine-tuning
  • Natural Language Processing course
  • University of Tehran · 2021
  • PyTorch · Hugging Face
BERT XLNet
0.62test F1, XLNet (BERT: 0.61)
0.97test recall, BERT: it misses almost no hateful tweet
0.44–0.49test precision: both models over-flag

Two ways to pretrain a transformer

BERT: masked language modelling

Hide 15% of the tokens and predict them from both sides at once. Bidirectional context makes BERT a strong sentence encoder; for classification, a linear head reads the [CLS] token.

XLNet: permutation language modelling

Predict tokens autoregressively, but over random factorisation orders, so each token learns from context on both sides without the artificial [MASK] token that never appears at fine-tuning time.

Results

BERT vs. XLNet

Interactive

Numbers from the repository's results table.

Both models lean hard toward flagging. Recall on the test split is 0.97 for BERT and 0.84 for XLNet, but precision is below one half: most posts they flag are not hateful. For moderation that can be a defensible operating point (a human reviews what is flagged), but it is not a finished classifier.

XLNet generalises a little better (test accuracy 0.56 vs. 0.48, F1 0.62 vs. 0.61), and both drop sharply from train to test, so the label distribution or vocabulary shifts between splits.

Next steps I would take: tune the decision threshold on a validation split instead of using 0.5, re-weight the classes, and report precision–recall curves rather than a single point.

BERT cost curve
BERT: cost
BERT performance curve
BERT: performance