Lectures

“In this introductory lecture I will give an overview of the course and we will discuss learning word and sentence representations from text.”

• Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.

• Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018.

• Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017.

“In this lecture, I will give an overview of the research projects.”

• Please see project descriptions and references to papers and resources on Canvas.

In this session we will discuss three recent papers on dependency-based word embeddings, specialisation and retro-fitting.

• In this session we will discuss the following papers:

• Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In proceedings of ACL 2014.

• Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy and Noah A. Smith. 2015. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of NAACL 2015.

• Douwe Kiela, Felix Hill and Stephen Clark. 2015. Specializing Word Embeddings for Similarity or Relatedness. In Proceedings of EMNLP 2015.

In this session we will discuss three recent papers on learning general-purpose sentence representations.

• In this session we will discuss the following papers:

• F Hill, K Cho, A Korhonen, Y Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics 4, 17-30

• Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of ICLR 2018.

In this session we will discuss contextualised word representations (ELMo).

• In this session we will discuss the following papers:

• Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer 2018. Deep contextualized word representations. In Proceedings of NAACL 2018, New Orleans, Louisiana.

• Allen Nie, Erin D. Bennett, Noah D. Goodman. 2018. DisSent: Learning Sentence Representations from Explicit Discourse Relations. arXiv.

In this session we will introduce attention and transformer architectures, and then discuss the recently proposed BERT model.

• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention Is All You Need. In Proceedings of NIPS 2017.

• A very helpful blog post explaining the transformer architecture.

• In this session we will discuss the following paper:

• Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of EMNLP 2018.

In this session we will discuss modelling ambiguity and metaphor.

• In this session we will discuss the following papers:

• Marek Rei, Luana Bulat, Douwe Kiela and Ekaterina Shutova. 2017. Grasping the Finer Point: A Supervised Similarity Network for Metaphor Detection. In Proceedings of EMNLP 2017, Copenhagen, Denmark.

• Ge Gao, Eunsol Choi, Yejin Choi and Luke Zettlemoyer. 2018. Neural Metaphor Detection in Context. In Proceedings of EMNLP 2018, Brussels, Belgium

In this session we will introduce the idea of multitask learning and the architectures proposed for it. We will then discuss the language processing tasks that can benefit each other through information sharing.

In this session we will discuss several recent multitask learning experiments.

• In this session we will discuss the following papers:

• Hashimoto, K., Tsuruoka, Y., & Socher, R. (2017). A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. In Proceedings of EMNLP 2017.

• Ma, J., Gao, W., & Wong, K. F. (2018). Detect rumor and stance jointly by neural multi-task learning. In Companion of the The Web Conference 2018.

In this session we will discuss learning representations of discourse fragments and documents.

• In this session we will discuss the following papers:

• Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, Eduard Hovy. 2016. Hierarchical Attention Networks for Document Classification. In Proceedings of NAACL 2016.

• Yangfeng Ji and Noah A. Smith. 2017. Neural Discourse Structure for Text Categorization. In Proceedings of ACL 2017.

In this session we will discuss learning multilingual word and sentence representations.

• In this session we will discuss the following papers:

• Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributed Semantics. In Proceedings of ACL 2014.

• Mikel Artetxe and Holger Schwenk. 2018. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. ArXiv.

In this session we will discuss interpretation of neural network models.

• In this session we will discuss the following paper:

• Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. Proceedings of ICLR 2019.

In this session we will discuss multimodal models, i.e. learning jointly from linguistic and visual data.

• A Comprehensive Survey of Deep Learning for Image Captioning

• In this session we will discuss the following papers:

• D. Kiela and L. Bottou 2014. Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. In Proceedings of EMNLP 2014, Doha, Qatar.

• Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of CVPR 2015.

In this session we will discuss the relationship of representation learning and neurocognition of language.

• A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain

• Natural speech reveals the semantic maps that tile human cerebral cortex

• In this session we will discuss the following papers:

• Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of CONLL 2018.

• Luana Bulat, Stephen Clark and Ekaterina Shutova. 2017. Speaking, Seeing, Understanding: Correlating semantics models with conceptual representation in the brain. In Proceedings of EMNLP 2017, Copenhagen, Denmark.

In this session, you will present the results of your research projects.