Lectures

“Introduction to the course”

• Chapter 4: Naive Bayes classification and sentiment in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss language models, i.e. modelling word sequences, using statistical techniques.”

• Required self-study: representing and estimating tabular Categorical distributions

• Optional, but highly recommended: it can be useful to learn a little bit about directed graphical models and tabular representation of conditional probability distributions. Check this video and slides

• Required - Section 3.2 (evaluation LMs) from Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Optional - Chapter 3: Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Optional (but recommended) - How to sample from a Categorical distribution: video or section 3.3 of textbook

“In this lecture, we will discuss labelling tasks such as part-of-speech tagging, named-entity recognition and semantic role labelling.”

• Required self-study: representing and estimating logistic Categorical distributions (part 1/2 theory, part 2/2 example)

• Required self-study: value recursion to compute mode and marginal probabilities in HMMs

• Play with the Colab example if you like

• Chapter 8: Part-of-speech tagging in Jurafsky and Martin (3rd edition).

“In this lecture, we will introduce syntax and ideas to model it.”

• Required Section 17.6 of textbook.

• Required Appendix C - statistical constituency parsing

• Recommended: check this video on dynamic programming algorithms for PCFGs.

• Chapter 17: Context-Free Grammars and Constituency Parsing in Jurafsky and Martin (3rd edition).

“In this lecture, we will introduce statistical models of word meaning, discuss generalisation from words to semantic classes and learning dense vector representations - word embeddings.”

• Chapter 23: Word Senses and WordNet in Jurafsky and Martin (3rd edition).

• Chapter 6: Vector semantics and embeddings in Jurafsky and Martin (3rd edition).

• A gentle introduction to neural networks can be found here

• The following paper provides a nice explanation of skip-gram with negative sampling: Yoav Goldberg and Omer Levy. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• This paper experiments with using syntactic dependency-based contexts to train the skip-gram model and provides some analysis on what kind of information the context vectors encode.

“In this lecture, we will discuss compositional semantics, i.e. modelling the meaning of phrases and sentences, and learning neural representations of sentences.”

• Chapter 7: Neural networks and neural language models in Jurafsky and Martin (3rd edition).

• Chapter 9: Sequence processing with recurrent neural networks in Jurafsky and Martin (3rd edition).

• A good and general reference for Neural Networks in NLP: Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing

• A gentle introduction to LSTMs is available here

• This is one of the papers that have introduced tree LSTM models: Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

“In this lecture, we will discuss discourse processing, i.e. modelling larger text fragments.”

• Chapter 26: Coreference resolution in Jurafsky and Martin (3rd edition).

• Chapter 27: Discourse coherence in Jurafsky and Martin (3rd edition).

“In this session, we will discuss large language models, their pretraining strategies, their architectures and their applications.”

• The slides list a number of papers that constitute the further reading material for this lecture.

• The Illustrated Transformer blog provides a very nice introduction to the Transformer architecture.

“We will discuss techniques that allow us to better understand what kind of information neural language models encode and how they do it.”

“In this lecture, we will talk about neural models of sequential data. Examples will include sequence labelling, machine translation (at a very introductory level; for much more, we will have an entire class dedicated to MT) and text summarisation.”

• An introduction to sequence-to-sequence models: Sequence to Sequence Learning with Neural Networks

• A survey of recent summarisation techniques is available here

“We will discuss the fundamentals of machine translation, including word-based / alignment models and phrase-based SMT”

• Neural Encoder-Decoder

• The Annotated Encoder-Decoder blog post

“In this lecture, we will introduce an important NLP application – dialogue modelling.”

• Chapter 15: Dialogue systems and chatbots in Jurafsky and Martin (3rd edition).

“In this lecture, we will give you a brief summary of the course and how different areas we have studied interface. We will also discuss what to expect at the exam.”