Lectures

“In this introductory lecture I will give an overview of the course and we will discuss learning word and sentence representations from text.”

• Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.

• Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018.

• Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017.

In this session we will introduce attention and transformer architectures

• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention Is All You Need. In Proceedings of NIPS 2017.

• A very helpful blog post explaining the transformer architecture.

• Visualization of attention heads for BERT: https://github.com/jessevig/bertviz

In this session we will discuss the BERT model.

• In this session we will discuss the following papers:

• Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL 2019.

• Ian Tenney, Dipanjan Das, Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACL 2019.

In this session we will discuss recent techniques for structured and unstructured pruning and finding task-specific subnetworks in Transformer models.

• Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. (2020). Gradient surgery for multi-task learning. arXivpreprint arXiv:2001.06782.

• In this session we will discuss the following papers:

• Paul Michel, Omer Levy, Graham Neubig. Are Sixteen Heads Really Better than One? In Proceedings of NeuroIPS 2019.

• Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin. The Lottery Ticket Hypothesis for Pre-trained BERT Networks. In Proceedings of NeuroIPS 2020.

In this session we will discuss learning multilingual word and sentence representations.

In this session we will discuss recent research on generative LMs, prompting, instruction-tuning and in-context learning.

• In this session we will discuss the following papers:

• Sanh et al., 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In Proceedings of ICLR 2022.

• Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models In Proceedings of NeurIPS 2022.

In this session we will discuss recent research on generative LMs, prompting, instruction-tuning and in-context learning.

• In this session we will discuss the following papers:

• Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Proceedings of EMNLP 2022.

• Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems.

In this guest lecture we will discuss safety and alignment of LLMs.

In this session we will discuss reseach on interpretability of LLM computations.

• In this session we will discuss the following papers:

• Mor Geva, Jasmijn Bastings, Katja Filippova, Amir Globerson. 2023. Dissecting Recall of Factual Associations in Auto-Regressive Language Models. In Proceedings of EMNLP 2023.

• Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West. 2024. Do Llamas Work in English? On the Latent Language of Multilingual Transformers. arXiv:2402.10588

In this session we will discuss research on bias in NLP models, diagnosing it and de-biasing.

• In this session we will discuss the following papers:

• David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Smith. 2023. ROBBIE: Robust Bias Evaluation of Large Generative Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3764–3814, Singapore. Association for Computational Linguistics.

• Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5108–5125, Singapore. Association for Computational Linguistics.

In this session we will discuss recent research on LLMs and their application beyond the English-speaking world.

• In this session we will discuss the following papers:

• Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, Daniel Hershcovich. 2023. Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study.

• Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.

In this session, we will discuss research on joint modelling of language and vision.

• In this session we will discuss the following papers:

• Wai Keen Vong, Wentao Wang, A. Emin Orhan, and Brenden M. Lake. 2024. Grounded language acquisition through the eyes and ears of a single child.

• Fu et al. 2024. BLINK: Multimodal Large Language Models Can See but Not Perceive.

In this session, you will present the results of your research projects.