Reading Course: ‘Natural Language Processing’

Overview

The purpose of this reading course is to develop students who can:

• engage in thoughtful, ethical, critique of Natural Language Processing (NLP);
• work productively to implement existing NLP methods; and
• use NLP to contribute to our understanding of the world.

Students are expected to develop:

• an understanding of NLP and its place in the world;
• exceptional written and verbal communication skills; and
• contribute in some small way to our understanding of something related to NLP.

Content

• Week 1: Essentials I
  • Silge, Julia & David Robinson, 2020, Text Mining with R, Chapters 1-4: https://www.tidytextmining.com.
  • Hovy, Dirk and Shannon L. Spruit, 2016, ‘The Social Impact of Natural Language Processing’, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 591–598,https://aclweb.org/anthology/P16-2096.pdf.
  • Prabhumoye, Shrimai, Elijah Mayfield, and Alan W Black, 2019, ‘Principled Frameworks for Evaluating Ethics in NLP Systems’, Proceedings of the 2019 Workshop on Widening NLP, https://aclweb.org/anthology/W19-3637/.
• Week 2: Essentials II
  • Silge, Julia & David Robinson, 2020, Text Mining with R, Chapters 5-7: https://www.tidytextmining.com.
  • Bolukbasi, Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama and Adam T. Kalai, 2016, ‘Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings’, Advances in Neural Information Processing Systems, 29 (NIPS 2016), http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-d.
  • Chang, Kai-Wei, Vinod Prabhakaran, and Vicente Ordonez, 2019, ‘Bias and Fairness in Natural Language Processing’, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial Abstracts, https://aclweb.org/anthology/D19-2004/.
• Week 3: Essentials III
  • Silge, Julia & David Robinson, 2020, Text Mining with R, Chapters 8-9: https://www.tidytextmining.com.
  • Hutchinson, Ben, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl, 2020, ‘Social Biases in NLP Models as Barriers forPersons with Disabilities’, arXiv, https://arxiv.org/abs/2005.00813.
• Week 4: NLP intermediate I
  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 1-3, https://smltar.com.
  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapter 3, https://web.stanford.edu/~jurafsky/slp3/.
  • Solaiman, Irene, Miles Brundage, Jack Clark, Amanda Askell, ArielHerbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, SarahKreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, Jasmine Wang, 2019, ‘Release Strategies and the Social Impacts of Language Models’,arXiv, https://arxiv.org/abs/1908.09203.
• Week 5: NLP intermediate II
  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 4-6, https://smltar.com.
  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 4 and 5, https://web.stanford.edu/~jurafsky/slp3/.
  • Tatman, Rachel, 2020, ‘What I Won’t Build’, Widening NLP Workshop 2020, Keynote address, 5 July, https://slideslive.com/38929585/what-i-wont-build and http://www.rctatman.com/talks/what-i-wont-build.
• Week 6: NLP intermediate III
  • Hvitfeldt, Emil & Julia Silge, 2020, Supervised Machine Learning for Text Analysis in R, Chapters 7-9, https://smltar.com.
  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 6 and 7, https://web.stanford.edu/~jurafsky/slp3/.
  • Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez and Kai-Wei Chang,2017, ‘Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints’, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2979–2989, https://aclweb.org/anthology/D17-1323.pdf.
• Week 7: Deep learning I
  • François Chollet with J. J. Allaire, 2018, Deep Learning with R, Chapters 1-4.
• Week 8: Deep learning II
  • François Chollet with J. J. Allaire, 2018, Deep Learning with R, Chapters 6.
  • Bender, Emily M. and Koller, Alexander, 2020, ‘Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data’, *Proceedings of the 58th Annual Meeting of the
• Week 9: Deep learning III
  • Anna Rogers, Isabelle Augenstein, 2020, ‘What Can We Do to Improve Peer Review in NLP?’, arXiv, 8 October, https://arxiv.org/abs/2010.03863.
  • Jurafsky, Dan, and James H. Martin, 2020, Speech and Language Processing, 3rd ed., Chapters 8 and 9, https://web.stanford.edu/~jurafsky/slp3/. Association for Computational Linguistics*, pp. 5185–5198, https://www.aclweb.org/anthology/2020.acl-main.463
• Week 10: Transformers I
  • Alammar, Jay, 2018, ‘The Illustrated Transformer’, http://jalammar.github.io/illustrated-transformer/.
  • Manning, Vaswani and Huang, 2019, ‘Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 14 – Transformers and Self-Attention’, https://www.youtube.com/watch?v=5vcj8kSwBCY&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z&index=14&ab_channel=stanfordonline.
  • Uszkoreit, Jakob, 2017, ‘Transformer: A Novel Neural Network Architecture for Language Understanding’, Google AI Blog, 31 August, https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html
• Week 11: Transformers II
  • Alammar, Jay, 2020, ‘How GPT3 Works - Visualizations and Animations’, 27 July, https://jalammar.github.io/how-gpt3-works-visualizations-animations/
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin, 2017, ‘Attention Is All You Need’, arXiv, http://arxiv.org/abs/1706.03762.
  • Jacob Devlin and Ming-Wei Chang, 2018, ‘Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing’, 2 November, Google AI Blog, https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html.
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2018, ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’, arXiv, https://arxiv.org/abs/1810.04805.
  • Rush, Alexander, 2018, ‘The Annotated Transformer’, https://nlp.seas.harvard.edu/2018/04/03/attention.html
• Week 12: Transformers III
  • Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei, 2020, ‘Language Models are Few-Shot Learners’, arXiv, https://arxiv.org/abs/2005.14165
  • (Fun/horrifying) Hao, Karen, 2020, ‘The messy, secretive reality behind OpenAI’s bid to save the world’, MIT Review, 17 February, https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/.

Assessment

• Learning Diary (15 per cent)
  • Date: Each week you will read relevant papers and books, engaging with them by writing notes and completing exercises. You will use GitHub to manage these notes and exercises and email a link to me at the end of each week.
• Presentation I (15 per cent)
  • Date: Roughly end of Week 4 (exact date determined by lab presentation cycle - it’ll be the end of a month and you’ll have at least a month’s notice).
  • Requirement: 10-15-minute presentation on what you’ve learned from Silge & Robinson.
• Presentation II (15 per cent)
  • Date: Roughly end of Week 8, as above.
  • Requirement: 10-15-minute presentation on an aspect of ethics, society, and NLP.
• Presentation III (15 per cent)
  • Date: Roughly end of Week 12, as above.
  • Requirement: 10-15-minute presentation on Hvitfeldt & Silge.
• Final Paper (40 per cent)
  • Toward the mid-term break we will have a meeting to discuss the topic of your final paper. It will be due on the last day of the exam period. This will be marked by me and reviewed by another DoSS professor.