Lectures

“Introduction to the course”

• Chapter 4: Naive Bayes classification and sentiment in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss language models, i.e. modelling word sequences, using statistical techniques.”

• Required self-study: representing and estimating tabular Categorical distributions

• Optional, but highly recommended: it can be useful to learn a little bit about directed graphical models and tabular representation of conditional probability distributions. Check this video and slides

• Required - Section 3.2 (evaluation LMs) from Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Optional - Chapter 3: Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Necessary for PnP1 - How to sample from a Categorical distribution: video or section 3.4 of textbook

“In this lecture, we will discuss labelling tasks such as part-of-speech tagging, named-entity recognition and semantic role labelling.”

• Required self-study: representing and estimating logistic Categorical distributions (part 1/2 theory, part 2/2 example)

• Required self-study: value recursion for HMMs

• Play with the Colab example if you like

• Chapter 17.2: Part-of-speech tagging in Jurafsky and Martin (3rd edition).

“In this lecture, we will introduce syntax and ideas to model it.”

• Required: Katia’s class on morphology (first 43 minutes of the video)

• Required: check this video on dynamic programming algorithms for PCFGs.

• Recommended Section 18.6 of textbook.

• Recommended Appendix C - statistical constituency parsing

“In this lecture, we will introduce statistical models of word meaning, discuss generalisation from words to semantic classes and learning dense vector representations - word embeddings.”

• Chapter G: Word Senses and WordNet in Jurafsky and Martin (3rd edition).

• Chapter 6: Vector semantics and embeddings in Jurafsky and Martin (3rd edition).

• A gentle introduction to neural networks can be found here

• The following paper provides a nice explanation of skip-gram with negative sampling: Yoav Goldberg and Omer Levy. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• This paper experiments with using syntactic dependency-based contexts to train the skip-gram model and provides some analysis on what kind of information the context vectors encode.

“In this lecture, we will discuss compositional semantics, i.e. modelling the meaning of phrases and sentences, and learning neural representations of sentences.”

• Chapter 7: Neural networks and neural language models in Jurafsky and Martin (3rd edition).

• Chapter 8: RNNs and LSTMs in Jurafsky and Martin (3rd edition).

• A good and general reference for Neural Networks in NLP: Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing

• A gentle introduction to LSTMs is available here

• This is one of the papers that have introduced tree LSTM models: Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

“In this lecture, we will discuss discourse processing, i.e. modelling larger text fragments.”

• Chapter 23: Coreference Resolution and Entity Linking in Jurafsky and Martin (3rd edition).

• Chapter 24: Discourse coherence in Jurafsky and Martin (3rd edition).

“In this session, we will discuss large language models, their pretraining strategies, their architectures and their applications.”

• The slides list a number of papers that constitute the further reading material for this lecture.

• The Illustrated Transformer blog provides a very nice introduction to the Transformer architecture.

“We will discuss techniques that allow us to better understand what kind of information neural language models encode and how they do it.”

“We will discuss techniques for making LLM outputs safe, including safety fine-tuning and alignment to human preferences.”

“In this lecture, we will introduce an important NLP application – dialogue modelling.”

• Chapter 15: Chatbots & Dialogue Systems in Jurafsky and Martin (3rd edition).

“In this lecture, we will talk about neural models of sequential data. Examples will include sequence labelling, machine translation and text summarisation.”

• An introduction to sequence-to-sequence models: Sequence to Sequence Learning with Neural Networks

“In this lecture, we will give you a brief summary of the course and how different areas we have studied interface. We will also discuss what to expect at the exam.”