What Schools Want vs. What Evidence Can Deliver | PIER Seminar

Value of the Information & Strength of the Argument

The talk provides a valuable critique of the current evidence system in education, highlighting the often-overlooked issue of treatment effect heterogeneity and its implications for external validity. Tipton’s argument is well-supported by references to her own research and other studies, such as the work by Eva Vivalt on researcher versus practitioner preferences. She effectively illustrates how the focus on average effects and homogeneous samples can mislead decision-makers. The argumentation is coherent and persuasive, though it is primarily based on the speaker’s expertise and interpretation rather than a systematic review of the literature.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor through the speaker’s expertise and the use of relevant research examples. Tipton references her own publications and those of colleagues, such as the ‘heterogeneity revolution’ paper with Chris Bryan and David Yeager. However, the talk does not provide a comprehensive list of sources, and the references are not systematically cited. The title accurately reflects the content, and the talk is well-structured. The audience’s questions and the speaker’s responses add to the credibility of the presentation.

The title accurately reflects the central tension discussed: what schools want (evidence for their specific context) versus what evidence systems typically deliver (average effects).

8/10

The talk is given by a professor of statistics and data science at Northwestern University, with expertise in field experiments and meta-analysis. The content is well-structured, references several studies and frameworks, and is presented in an academic setting. However, as a seminar, it is largely an opinion piece based on the speaker's experience and interpretation of existing research, not a systematic review or original study.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 03:00Introduction of the tension: schools ask 'will this work in my school?' but evidence systems answer 'does it work on average?'
• 10:00Discussion of the implicit assumption of constant treatment effects in clearinghouses and meta-analyses.
• 20:00Explanation of how study design often reduces variation, leading to homogeneous samples that limit external validity.
• 30:00Presentation of the 'heterogeneity revolution' concept and the importance of who is in the sample.
• 40:00Introduction of the 'engineering mindset' and the proposal to assume treatment effects vary until proven otherwise.
• 50:00Discussion of the difference between scientific and engineering questions, and the need for evidence designed for specific contexts.
• 1:00:00Example from Nicole Patton-Terry's work on reading research illustrating the application of the engineering mindset.

• PIER Seminar — Seminar series page
• Harvard Graduate School of Education — Institution website
• HGSE LinkedIn — Social media page
• HGSE Threads — Social media page

• What Works Clearinghouse — The clearinghouse is discussed as an example of the current evidence system.