Lectures

“In this introductory lecture I will give an overview of the course and we will discuss learning word and sentence representations from text.”

• Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.

• Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018.

• Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017.

In this session we will discuss three recent papers on dependency-based word embeddings, specialisation and retro-fitting.

• In this session we will discuss the following papers:

• Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In proceedings of ACL 2014.

• Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy and Noah A. Smith. 2015. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of NAACL 2015.

• Douwe Kiela, Felix Hill and Stephen Clark. 2015. Specializing Word Embeddings for Similarity or Relatedness. In Proceedings of EMNLP 2015.

In this session we will discuss two recent papers on learning general-purpose sentence representations.

• Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of ICLR 2018.

• Please see Canvas for project descriptions and references to related papers.

• In this session we will discuss the following papers:

• F Hill, K Cho, A Korhonen, Y Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics 4, 17-30

• Allen Nie, Erin D. Bennett, Noah D. Goodman. 2019. DisSent: Learning Sentence Representations from Explicit Discourse Relations. In Proceedings of ACL 2019.

In this session we will discuss contextualised word representations (ELMo) and how they can be applied to model meaning variation in context.

• In this session we will discuss the following papers:

• Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer 2018. Deep contextualized word representations. In Proceedings of NAACL 2018, New Orleans, Louisiana.

• Ge Gao, Eunsol Choi, Yejin Choi and Luke Zettlemoyer. 2018. Neural Metaphor Detection in Context. In Proceedings of EMNLP 2018, Brussels, Belgium

In this session we will introduce attention and transformer architectures, and then discuss the recently proposed BERT model.

• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention Is All You Need. In Proceedings of NIPS 2017.

• A very helpful blog post explaining the transformer architecture.

• Visualization of attention heads for BERT: https://github.com/jessevig/bertviz

• In this session we will discuss the following paper:

• Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL 2019.

In this session we will introduce the idea of multitask learning and the architectures proposed for it. We will then discuss the language processing tasks that can benefit each other through information sharing.

In this session we will discuss several recent multitask learning experiments.

• In this session we will discuss the following papers:

• Hashimoto, K., Tsuruoka, Y., & Socher, R. (2017). A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. In Proceedings of EMNLP 2017.

• Verna Dankers, Marek Rei, Martha Lewis and Ekaterina Shutova (2019). Modelling the interplay of metaphor and emotion through multitask learning. In Proceedings of EMNLP 2019.

In this session I will introduce the meta-learning framerwork and we will discuss the application of meta-learning to NLP.

• The paper by Finn et al (2017) introducing model-agnostic meta-learning.

• In this session we will discuss the following papers:

• Trapit Bansal, Rishikesh Jha, Andrew McCallum, 2019. Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks. Arxiv.

In this session we will discuss learning multilingual word and sentence representations.

• In this session we will discuss the following papers:

• Mikel Artetxe and Holger Schwenk. 2018. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics.

• Nooralahzadeh, F., Bekoulis, G., Bjerva, J., & Augenstein, I. (2020). Zero-Shot Cross-Lingual Transfer with Meta Learning. arXiv preprint arXiv:2003.02739.

In this session we will discuss learning representations of discourse fragments and documents, and modelling different types of discourse structure: rhetorical relations and argumentation.

• Please find the presentation of the AMPERSAND paper here.

• In this session we will discuss the following papers:

• Yangfeng Ji and Noah A. Smith. 2017. Neural Discourse Structure for Text Categorization. In Proceedings of ACL 2017.

• Tuhin Chakrabarty, Christopher Hidey, Smaranda Muresan, Kathy McKeown, Alyssa Hwang. 2019. AMPERSAND: Argument Mining for PERSuAsive oNline Discussions. In Proceedings of EMNLP 2019.

In this session we will discuss the relationship of representation learning and neurocognition of language.

• A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain

• Natural speech reveals the semantic maps that tile human cerebral cortex

• In this session we will discuss the following papers:

• Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of CONLL 2018.

• Ian Tenney, Dipanjan Das, Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACL 2019.

In this session, you will present the results of your research projects.