Toronto Workshop on Reproducibility

An annual workshop focused on reproducibility in data science and statistics. First held 25-26 February 2021, and again 23-25 February 2022. Free and hosted via Zoom. Jointly hosted by CANSSI Ontario and the Data Sciences Institute. Supported by the Faculty of Information and the Department of Statistical Sciences.

Overview

This conference brings together academic and industry participants on the critical issue of reproducibility in applied statistics and related areas. The conference is free and hosted online. Everyone is welcome, you don’t need to be affiliated with a university.

The conference has three broad areas of focus:

2022

Timetable

Wednesday, 23 February 2022

Time Speaker Talk
08:40-09:00 Lisa Strug, University of Toronto Introduction and welcome
09:00-09:30 Benjamin Haibe-Kains, University Health Network The (Not-So-)Hard Path To Transparency and Reproducibility in AI Research
09:30-10:00 Colm-cille Caulfield, University of Cambridge Reproducibility in an Uncertain World: How should academic data science researchers give advice?
10:00-10:30 Stephen Eglen, University of Cambridge Evaluating the reproducibility of computational results reported in scientific journals
10:30-11:00 Valentin Danchev, University of Essex Reproducibility and Replicability of Large Pre-trained Language Models
11:00-11:30 Monica Alexander, University of Toronto Reproducibility in Demography: where are we at and where can we go?
11:30-12:00 Break
12:00-12:30 Ariel Mundo, University of Arkansas Statistics and reproducibility in biomedical research: Why we need both
12:30-13:00 Shilaan Alzahawi, Stanford University Lay perceptions of scientific findings: Swayed by the crowd?
13:00-13:30 Break
13:30-14:00 Fernando Hoces de la Guardia, University of California, Berkeley Social Sciences Reproducibility Platform
14:00-15:30 Break
15:30-16:00 Carl Laflamme, YCharOS Antibody Characterization through Open Science (YCharOS)
16:00-16:30 Robert Hanisch, National Institute of Standards and Technology and Research Data Alliance Reproducibility: A Metrology Perspective
16:30-17:00 Yann Joly, McGill University Incentivizing open data sharing - what’s in it for me!?

Thursday, 24 February 2022

Time Speaker Talk
08:30-09:00 Julien Chiquet, Université Paris-Saclay Computo: a journal of the French Statistical Society promoting reproductibility
09:00-09:30 Nick Radcliffe, Global Open Finance Centre at the University of Edinburgh Gentest: Automatic Test Generation for Data Science
09:30-10:00 Markus Fritsch, University of Passau Towards reproducible GMM estimation
10:00-10:30 Break
10:30-11:00 Aneta Piekut, Sheffield Methods Institute, University of Sheffield Integrating reproducibility into the curriculum of an undergraduate social sciences degree
11:00-12:30 Break
12:30-13:00 Jason Hattrick,-Simpers, University of Toronto Towards Trust and Reproducibility in Materials AI
13:00-13:30 Aya Mitani, University of Toronto Reproducible, reliable, replicable? In-class exercise using peer-reviewed studies
13:30-14:00 Shannon Ellis, UC San Diego Structuring & Managing Group Projects in Large-Enrollment Undergraduate Data Science Courses
14:00-14:30 Maria Tackett, Duke University Knit, Commit, and Push: Teaching version control in undergraduate statistics courses
14:30-15:00 Break
15:00-15:30 Lars Vilhuber, Cornell University Teaching for large-scale Reproducibility Verification
15:30-16:00 Michael Geuenich, Lunenfeld Tanenbaum Research Institute and University of Toronto With great data come great pipelines: creating flexible standardized pipelines for common biomedical analysis tasks using Snakemake
16:00-16:30 Paraskevi Massara, University of Toronto MOSS4Research: A maturity model to evaluate and improve reproducibility in research projects
16:30-17:00 Chris Kenny, Harvard University Reproducible Redistricting
17:00-17:30 Dewi Amaliah, Monash University Reproducible Practice in Taming the Wild Data

Friday, 25 February 2022

Time Speaker Talk
09:00-09:30 Marco Prado, University of Western Ontario Reproducibility for Behavior Experiments in Basic Science
09:30-10:00 David Grubbs and Lara Spieker, CRC Press On book publishing
10:00-11:00 Joelle Pineau, McGill University & Meta (Facebook) AI Research Improving Reproducibility in Machine Learning Research
11:00-11:30 Debbie Yuster, Ramapo College of New Jersey Infusing Reproducibility into Introductory Data Science
11:30-12:00 Colin Rundel, Duke University Teaching Statistical computing with Git and GitHub
12:00-12:30 Mine Çetinkaya,-Rundel, Duke University and RStudio Reproducible authoring with Quarto
12:30-13:00 Erin Heerey, Western University The Experimenter in the Room
13:00-13:30 John McLevey, University of Waterloo Reproducibility and Principled Data Processing in Python
13:30-14:00 Break
14:00-14:30 Kevin Wilson, Brown University and Jake Bowers, University of Illinois at Urbana-Champaign Six Tips for Reproducible Field Experiments
14:30-15:00 Abel Brodeur, University of Ottawa Introducing the Institute for Replication
15:00-15:30 Allison Koenecke, Cornell University and Microsoft Research Reproducible Retrospective Analysis
15:30-16:30 Michael Hoffman, University Health Network and University of Toronto Reproducibility standards for machine learning in the life sciences

Presenter biographies and abstracts

Keynotes

Invited talks

2021

Timetable

Thursday, 25 February, 2021

Time Speaker Focus Recording
9:00-9:10am Rohan Alexander, University of Toronto Welcome -
9:10-9:20am Radu Craiu, University of Toronto Opening remarks https://youtu.be/JGGVEgMBURU
9:20-9:30am Wendy Duff, University of Toronto Opening remarks https://youtu.be/Z3aWU1A0FCw
9:30-10:25am Mine Çetinkaya-Rundel, University of Edinburgh Keynote - Teaching https://youtu.be/ANH2tv2vkew
10:30-11:30am Riana Minocher, Max Planck Institute for Evolutionary Anthropology Keynote - Evaluating https://youtu.be/O3t8TwWeli0
11:30-11:55am Tiffany Timbers, University of British Columbia Teaching https://youtu.be/mh93W8XimOg
Noon-12:25pm Tyler Girard, University of Western Ontario Teaching https://youtu.be/k3qgmUAjIvA
12:30-12:55pm Shiro Kuriwaki, Harvard University Practices https://youtu.be/-J-eiPnmoNE
1:00-1:25pm Meghan Hoyer, Washington Post & Larry Fenn AP Practices https://youtu.be/FFwMfNk83rc
1:30-1:55pm Tom Barton, Royal Holloway, University of London Evaluating https://youtu.be/YTdhcSDqFNQ
2:00-2:25pm Break - -
2:30-2:55pm Mauricio Vargas, Catholic University of Chile & Nicolas Didier Arizona State University Evaluating https://youtu.be/VpTavLYEMgg
3:00-3:25pm Jake Bowers, University of Illinois & The Policy Lab Practices https://youtu.be/3N0YwJIbbHg
3:30-3:55pm Amber Simpson, Queens University Practices https://youtu.be/uUfrcB6aynQ
4:00-4:25pm Garret Christensen, US FDIC Evaluating https://youtu.be/595KkVKJ29w
4:30-4:55pm Yanbo Tang, University of Toronto Practices https://youtu.be/0x6gOkldOvk
5:00-5:25pm Lauren Kennedy, Monash University Practices https://youtu.be/HhfogRbgbA4
5:30-6:00pm Lisa Strug, University of Toronto & CANSSI Ontario Closing remarks https://youtu.be/B_9puTSp3f8

Friday, 26 February, 2021

Time Speaker Focus Recording
8:00-8:30am Nick Radcliffe and Pei Shan Yu, Global Open Finance Centre of Excellence & University of Edinburgh Practices https://youtu.be/pWEc8XoIIKE
8:30-9:00am Julia Schulte-Cloos, LMU Munich Practices -
9:00-9:25am Simeon Carstens, Tweag/IO Practices https://youtu.be/fpoFzDvrJAA
9:30-9:55am Break - -
10:00-10:55am Eva Vivalt, University of Toronto Keynote - Practices https://youtu.be/0WZUzf2oSGY
11:00-11:25am Andrés Cruz, Pontificia Universidad Católica de Chile Practices https://youtu.be/HjdPDEACxmA
11:30-11:55am Emily Riederer, Capital One Practices https://youtu.be/BknQ0ZNkMNY
Noon-12:25pm Florencia D’Andrea, National Institute of Agricultural Technology Practices https://youtu.be/9FVUIPfBeXw
12:30-12:55pm John Blischak, Freelance scientific software developer Practices https://youtu.be/RrcaGukYDyE
1:00-1:25pm Shemra Rizzo, Genentech Practices https://youtu.be/rEYtB3CG76Q
1:30-2:25pm Break - -
2:30-2:55pm Wijdan Tariq, University of Toronto Evaluating -
3:00-3:25pm Sharla Gelfand, Freelance R Developer Practices https://youtu.be/G5Nm-GpmrLw
3:30-3:55pm Ryan Briggs, University of Guelph Practices https://youtu.be/_dgGbxItiB4
4:00-4:25pm Monica Alexander, University of Toronto Practices https://youtu.be/yvM2C6aZ94k
4:30-4:55pm Annie Collins, University of Toronto Practices https://youtu.be/u4ibhN_nWyI
5:00-5:25pm Nancy Reid, University of Toronto Practices https://youtu.be/sIsOPuZOQL4
5:30-6:00pm Rohan Alexander, University of Toronto Closing remarks https://youtu.be/7LttFNOI6p8

Presenter biographies and abstracts

Keynotes:

Invited presentations:

Code of conduct

Code

The organizers of the Toronto Workshop on Reproducibility are dedicated to providing a harassment-free experience for everyone regardless of age, gender, sexual orientation, disability, physical appearance, race, or religion (or lack thereof).

All participants (including attendees, speakers, sponsors and volunteers) at the Toronto Workshop on Reproducibility are required to agree to the following code of conduct.

The code of conduct applies to all conference activities including talks, panels, workshops, and social events. It extends to conference-specific exchanges on social media, for instance posts tagged with the identifier of the conference (e.g. #TOrepro on Twitter), and replies to such posts.

Organizers will enforce this code throughout and expect cooperation in ensuring a safe environment for all.

Expected Behaviour

All conference participants agree to:

Unacceptable Behaviour

Behaviour that is unacceptable includes, but is not limited to:

If you are asked to stop harassing behaviour you should stop immediately, even if your behaviour was meant to be friendly or a joke, it was clearly not taken that way and for the comfort of all conference attendees you should stop.

Attendees who behave in a manner deemed inappropriate are subject to actions listed under ‘Procedure for Code of Conduct Violations’.

Additional Requirements for Conference Contributions

Presentation slides and posters should not contain offensive or sexualised material. If this material is impossible to avoid given the topic (for example text mining of material from hate sites) the existence of this material should be noted in the abstract and, in the case of oral contributions, at the start of the talk or session.

Procedure for Code of Conduct Violations

The organizing committee reserves the right to determine the appropriate response for all code of conduct violations. Potential responses include:

What To Do If You Witness or Are Subject To Unacceptable Behaviour

If you are being harassed, notice that someone else is being harassed, or have any other concerns relating to harassment, please contact Rohan Alexander - , or Kelly Lyons - .

We will take all good-faith reports of harassment by Toronto Workshop on Reproducibility participants seriously.

We reserve the right to reject any report we believe to have been made in bad faith. This includes reports intended to silence legitimate criticism.

We will respect confidentiality requests for the purpose of protecting victims of abuse. We will not name harassment victims without their affirmative consent.

Questions or concerns about the Code of Conduct can be addressed to .

Acknowledgements

Parts of the above text are licensed CC BY-SA 4.0. Credit to SRCCON. This code of conduct was based on that developed for useR! 2018 which was a revision of the code of conduct used at previous useR!s and also drew from rOpenSci’s code of conduct.