← All projects

Sentence Generation with an LSTM Language Model

A recurrent network that reads a sentence one word at a time and predicts the next, then writes new sentences by sampling from itself. The real question was which knobs matter, so I changed them one at a time.

  • Language modelling
  • Natural Language Processing course
  • University of Tehran ยท 2021
  • PyTorch
LSTM language model banner
8models in four paired experiments
1hyperparameter changed per pair, all else fixed
50epochs each, batch 256
Perplexityplus the quality of generated sentences

The model

  1. EmbedEach word index becomes a dense vector (50 or 200 dimensions).
  2. RecurStacked LSTM layers carry a hidden state through the sentence, gating what to keep and forget.
  3. PredictA softmax over the vocabulary gives P(next word | words so far); training minimises cross-entropy, i.e. log-perplexity.
  4. GenerateSample a word, feed it back in, repeat until the end-of-sentence token.

Which hyperparameter matters?

Pick an experiment

Interactive
Sentences generated by the two models