Lectures

“Introduction to the course”

• Chapter B: Naive Bayes classification and sentiment in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss language models, i.e. modelling word sequences, using statistical techniques.”

• Required self-study: representing and estimating tabular Categorical distributions - on Canvas Media Gallery or YouTube

• Optional, but highly recommended: it can be useful to learn a little bit about directed graphical models and tabular representation of conditional probability distributions. Check this video (also available on Canvas Media Gallery) and slides

• Required - Section 3.2 (evaluation LMs) from Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Optional - Chapter 3: Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Necessary for PnP1 - How to sample from a Categorical distribution: on Canvas Media Gallery or YouTube or section 3.4 of textbook

“In this lecture, we will discuss labelling tasks such as part-of-speech tagging, named-entity recognition and semantic role labelling.”

• Required self-study: representing and estimating logistic Categorical distributions (Canvas Media Gallery or YouTube - part 1/2 theory, YouTube - part 2/2 example)

• Required self-study: value recursion for HMMs on Canvas Media Gallery or YouTube

• Play with the Colab example if you like

• Chapter 17.2: Part-of-speech tagging in Jurafsky and Martin (3rd edition).

“In this lecture, we will introduce syntax and ideas to model it.”

• Required: Katia’s class on morphology (first 43 minutes of the video)

• Required: check video on dynamic programming algorithms for PCFGs on Canvas Media Gallery or YouTube.

• Recommended Section 18.6 of textbook.

• Recommended Chapter E - statistical constituency parsing

“In this lecture, we will introduce statistical models of word meaning, discuss generalisation from words to semantic classes and learning dense vector representations - word embeddings.”

• Chapter I: Word Senses and WordNet in Jurafsky and Martin (3rd edition).

• Chapter 5: Vector semantics and embeddings in Jurafsky and Martin (3rd edition).

• A gentle introduction to neural networks can be found here

• The following paper provides a nice explanation of skip-gram with negative sampling: Yoav Goldberg and Omer Levy. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• This paper experiments with using syntactic dependency-based contexts to train the skip-gram model and provides some analysis on what kind of information the context vectors encode.

“In this lecture, we will discuss compositional semantics, i.e. modelling the meaning of phrases and sentences, and learning neural representations of sentences.”

• Chapter 6: Neural networks in Jurafsky and Martin (3rd edition).

• Chapter 13: RNNs and LSTMs in Jurafsky and Martin (3rd edition).

• A good and general reference for Neural Networks in NLP: Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing

• A gentle introduction to LSTMs is available here

• This is one of the papers that have introduced tree LSTM models: Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

“In this lecture, we will discuss discourse processing, i.e. modelling larger text fragments.”

• Chapter 23: Coreference Resolution and Entity Linking in Jurafsky and Martin (3rd edition).

• Chapter 24: Discourse coherence in Jurafsky and Martin (3rd edition).

“In this lecture, we will talk about neural models of sequential data. Examples will include sequence labelling, machine translation and text summarisation.”

• An introduction to sequence-to-sequence models: Sequence to Sequence Learning with Neural Networks

“In this session, we will discuss large language models, their pretraining strategies, their architectures and their applications.”

• The slides list a number of papers that constitute the further reading material for this lecture.

• The Illustrated Transformer blog provides a very nice introduction to the Transformer architecture.

“In this lecture, we will discuss good practices in LLM development.”

“We will discuss techniques for making LLM outputs safe, including safety fine-tuning and alignment to human preferences.”

“In this lecture, we will discuss LLM use in practice, i.e. various prompting strategies and techniques for parameter efficient fine-tuning, and then talk about LLM agents and their memory.”

• The slides list a number of papers that constitute the further reading material for this lecture.

“We will discuss techniques that allow us to better understand what kind of information neural language models encode and how they do it.”