Lectures

“Introduction to the course”

• Chapter 4: Naive Bayes classification and sentiment in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss morphological processing”

• Lecture notes are available here

“In this lecture, we will discuss language models, i.e. modelling word sequences, and part-of-speech tagging”

• Chapter 3: Language modelling with n-grams in Jurafsky and Martin (3rd edition).

• Chapter 8: Part-of-speech tagging in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss syntax and syntactic parsing”

• Chapter 10: Formal grammars of English in Jurafsky and Martin (3rd edition).

• Chapter 11: Syntactic parsing in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss lexical semantics, i.e. modelling the meaning of words, and will introduce statistical models of word meaning”

• Appendix Chapter C: Computing with Word Senses: WSD and WordNet in Jurafsky and Martin (3rd edition).

• Chapter 6: Vector semantics in Jurafsky and Martin (3rd edition).

“In this lecture, we will discuss generalisation from words to semantic classes and learning dense vector representations - word embeddings.”

• Chapter 6: Vector semantics in Jurafsky and Martin (3rd edition).

• A gentle introduction to neural networks can be found here

“In this lecture, we will discuss methods for learning dense word embeddings and compositional semantics, i.e. modelling the meaning of phrases and sentences, and learning neural representations of sentences.”

• Chapter 6: Vector semantics in Jurafsky and Martin (3rd edition) has a section on word embeddings.

• Chapter 7: Neural networks and neural language models in Jurafsky and Martin (3rd edition).

• Chapter 9: Sequence processing with recurrent neural networks in Jurafsky and Martin (3rd edition).

• The following paper provides a nice explanation of skip-gram with negative sampling: Yoav Goldberg and Omer Levy. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• A good and general reference for Neural Networks in NLP: Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing

“In this lecture, we will finish our discussion of compositional semantics and then talk about discourse processing, i.e. modelling larger text fragments.”

• A gentle introduction to LSTMs is available here

• This is one of the papers that have introduced tree LSTM models: Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

“In this lecture, we will discuss modelling textual entailment and paraphrasing methods.”

• A Survey of Paraphrasing and Textual Entailment Methods (Sections 1 and 2)

• A large annotated corpus for learning natural language inference

• Annotation Artifacts in Natural Language Inference Data

“In this lecture, we will discuss dialogue modelling and dialogue systems.”

• Elia Bruni’s slides are available here

• Chapter 24 of Jurafsky & Martin (3rd edition): Dialog Systems and Chatbots

• Chapter 25 of J&M (3rd ed): Advanced Dialog Systems

• A Survey of Available Corpora for Building Data-Driven Dialogue Systems

• Section 2 of this paper offers a concise overview of data-driven dialogue systems

• Tutorial on Deep Learning for Dialogue Systems at COLING 2018

• GuessWhat?! Visual object discovery through multi-modal dialogue

• Visual Dialog

“In this lecture, we will talk about language generation and cover a particular language generation task, text summarisation, in more detail.”

• A survey of recent summarisation techniques is available here

• An introduction to sequence-to-sequence models: Sequence to Sequence Learning with Neural Networks

“We will discuss the fundamentals of machine translation, incl. word-based / alignment models and phrase-based SMT”

• Explanation of IBM models 1 and 2

• Optional readings

• Word based IBM models

• Phrase-based SMT and this book

• Neural Encoder-Decoder

• The Annotated Encoder-Decoder blog post

In this lecture, we will discuss the differences between frequentism and Bayesian modelling. We will discuss the concept of a prior and Bayesian inference. The model we will use to illustrate concepts is the Dirichlet-Multinomial model, the base for models such as Bayesian mixture models, HMM, and LDA. For approximate inference, we will discuss MCMC and in particular Gibbs sampling.

Shameless advertisement: * To learn more about latent variable models in NLP (unsupervised and semi-supervised learning), take NLP2. * To learn more about Bayesian modelling in NLP, especially in the context of deep learning, take Machine Learning for NLP ;)

• • For a POS tagging model: A Fully Bayesian Approach to Unsupervised Part-of-Speech Tagging
  • For a PCFG model: Bayesian Inference for PCFGs via Markov chain Monte Carlo
  • A very special type of mixture model: Bayesian Word Alignment for Statistical Machine Translation
  • The classict of all times: Latent Dirichlet Allocation
    • This paper focusses on a different type of approximate inference technique (not MCMC, but rather variational inference), in ML4NLP we cover it in great detail (in particular, this is the class of algorithms we use to do probabilistic modelling with neural networks)
  • If you are interested in Bayesian non-parametric methods for NLP, check Sharon Goldwater’s thesis, it’s remarkably well written and clear!