Natural Language Processing

Overview

The purpose of this course is to develop students who can:

  • engage in thoughtful, ethical, critique of Natural Language Processing (NLP), especially Large Language Models (LLMs);
  • work productively to implement existing NLP methods, especially LLMs; and
  • use NLP, and especially LLMs, to contribute to our understanding of the world.

Students are expected to develop:

  • an understanding of NLP, especially LLMs, and its place in the world;
  • exceptional written and verbal communication skills; and
  • contribute in some small way to our understanding of something related to NLP, ideally LLMs.

Pre-requisites

  • Comfortable coding in R and Python and using GitHub.
  • Go through Silge, Julia & David Robinson, 2020, Text Mining with R, https://www.tidytextmining.com.

Content

Week 1: Essentials I

  • (Motivation) Karpathy, Andrej, 2013, “Intro to Large Language Models”, YouTube, 22 November, https://youtu.be/zjkBMFhNj_g.
  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 1-3, https://smltar.com.
  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapter 3, https://web.stanford.edu/~jurafsky/slp3/.
  • Hovy, Dirk and Shannon L. Spruit, 2016, ‘The Social Impact of Natural Language Processing’, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 591–598,https://aclweb.org/anthology/P16-2096.pdf.
  • Prabhumoye, Shrimai, Elijah Mayfield, and Alan W Black, 2019, ‘Principled Frameworks for Evaluating Ethics in NLP Systems’, Proceedings of the 2019 Workshop on Widening NLP, https://aclweb.org/anthology/W19-3637/.

Week 2: Essentials II

  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 4-6, https://smltar.com.
  • Bolukbasi, Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama and Adam T. Kalai, 2016, ‘Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings’, Advances in Neural Information Processing Systems, 29 (NIPS 2016), http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-d.
  • Chang, Kai-Wei, Vinod Prabhakaran, and Vicente Ordonez, 2019, ‘Bias and Fairness in Natural Language Processing’, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial Abstracts, https://aclweb.org/anthology/D19-2004/.

Week 3: Essentials III

  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 4 and 5, https://web.stanford.edu/~jurafsky/slp3/.
  • Hutchinson, Ben, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl, 2020, ‘Social Biases in NLP Models as Barriers forPersons with Disabilities’, arXiv, https://arxiv.org/abs/2005.00813.

Week 4: NLP intermediate I

  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 6 and 7, https://web.stanford.edu/~jurafsky/slp3/.
  • Solaiman, Irene, Miles Brundage, Jack Clark, Amanda Askell, ArielHerbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, SarahKreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, Jasmine Wang, 2019, ‘Release Strategies and the Social Impacts of Language Models’,arXiv, https://arxiv.org/abs/1908.09203.

Week 5: NLP intermediate II

  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 7-9, https://smltar.com.
  • Tatman, Rachel, 2020, ‘What I Won’t Build’, Widening NLP Workshop 2020, Keynote address, 5 July, https://slideslive.com/38929585/what-i-wont-build and http://www.rctatman.com/talks/what-i-wont-build.

Week 6: NLP intermediate III

  • François Chollet, 2021, Deep Learning with Python, Chapters 1-4.
  • Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez and Kai-Wei Chang,2017, ‘Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints’, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2979–2989, https://aclweb.org/anthology/D17-1323.pdf.

Week 7: Deep learning I

  • François Chollet, 2021, Deep Learning with Python, Chapters 5-7.

Week 8: Deep learning II

  • François Chollet, 2021, Deep Learning with Python, Chapters 11 and 12.
  • Bender, Emily M. and Koller, Alexander, 2020, ‘Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data’, *Proceedings of the 58th Annual Meeting of the

Week 9: Deep learning III

  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 8 and 9, https://web.stanford.edu/~jurafsky/slp3/. Association for Computational Linguistics*, pp. 5185–5198, https://www.aclweb.org/anthology/2020.acl-main.463
  • Anna Rogers, Isabelle Augenstein, 2020, ‘What Can We Do to Improve Peer Review in NLP?’, arXiv, 8 October, https://arxiv.org/abs/2010.03863.

Week 10: Transformers I

  • Karpathy, Andrej, 2022, Neural Networks: Zero to Hero, Both of ‘The spelled-out intro…’ videos.
  • Alammar, Jay, 2018, ‘The Illustrated Transformer’, http://jalammar.github.io/illustrated-transformer/.
  • Boykis, Vicky, What are embeddings?, https://vickiboykis.com/what_are_embeddings/
  • Manning, Vaswani and Huang, 2019, ‘Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 14 – Transformers and Self-Attention’, https://www.youtube.com/watch?v=5vcj8kSwBCY&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z&index=14&ab_channel=stanfordonline.
  • Uszkoreit, Jakob, 2017, ‘Transformer: A Novel Neural Network Architecture for Language Understanding’, Google AI Blog, 31 August, https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html

Week 11: Transformers II

  • Karpathy, Andrej, 2022, Neural Networks: Zero to Hero, The ‘Building makemore…’ videos
  • Alammar, Jay, 2020, ‘How GPT3 Works - Visualizations and Animations’, 27 July, https://jalammar.github.io/how-gpt3-works-visualizations-animations/
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin, 2017, ‘Attention Is All You Need’, arXiv, http://arxiv.org/abs/1706.03762.
  • Jacob Devlin and Ming-Wei Chang, 2018, ‘Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing’, 2 November, Google AI Blog, https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html.
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2018, ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’, arXiv, https://arxiv.org/abs/1810.04805.
  • Rush, Alexander, 2018, ‘The Annotated Transformer’, https://nlp.seas.harvard.edu/2018/04/03/attention.html

Week 12: Transformers III

  • Karpathy, Andrej, 2022, Neural Networks: Zero to Hero, The ‘Let’s build GPT: from scratch, in code, spelled out.’
  • Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei, 2020, ‘Language Models are Few-Shot Learners’, arXiv, https://arxiv.org/abs/2005.14165
  • (Fun/horrifying) Hao, Karen, 2020, ‘The messy, secretive reality behind OpenAI’s bid to save the world’, MIT Review, 17 February, https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/.

Assessment

Notebook

  • Due date: Try to keep this updated weekly, on average, over the course of the term.
  • Task: Use Quarto to keep a notebook of what you read in the style of this one by Andrew Heiss.
  • Weight: 25 per cent.

Paper #1

  • Due date: Thursday, noon, Week 1.
  • Task: Donaldson Paper (although vary the dataset, so that it is something related to NLP).
  • Weight: 10 per cent.

Paper #2

  • Due date: Thursday, noon, Week 6.
  • Task: Write a paper applying what you are learning.
  • Weight: 25 per cent.

Final Paper

  • Due date: Thursday, noon, Week 12 + two weeks.
  • Task: Write a paper that involves doing original research.
  • Weight: 40 per cent.