Education Policy Speaker Series: Joseph Cimpian

“Large-Scale Estimates of LGBQ–Heterosexual Disparities in the Presence of Potentially Mischievous Responders: A Pre-Registered Replication and Comparison of Methods”

Although numerous survey-based studies have found that students who identify as lesbian, gay, bisexual, or questioning (LGBQ) have elevated risk for many negative academic, disciplinary, psychological, and health outcomes, the validity of the types of data on which these results rest have come under increased scrutiny. Over the past several years, a variety of data-validity screening techniques have been employed in attempts to scrub datasets of “mischievous responders,” youths who systematically provide extreme and untrue responses to outcome items and who tend to falsely report being LGBQ. We conducted a pre-registered replication of Cimpian et al. (2018) with the 2017 Youth Risk Behavior Survey (YRBS) to (1) estimate new LGBQ–heterosexual disparities on 20 outcomes; (2) test a broader, mechanistic theory relating mischievousness effects with a feature of items (i.e., item response-option extremity); and (3) compare four techniques used to address mischievous responders. Our results are consistent with Cimpian et al.’s findings that potentially mischievous responders inflate LGBQ–heterosexual disparities, do so more among boys than girls, and affect outcomes differentially. We also find that the method used to address mischievousness can lead to differences in LGBQ–heterosexual disparities of statistical and practical significance, with boosted regressions coupled with data removal leading to the smallest LGBQ–heterosexual disparities. The consistent patterns of results across several large-scale, pre-registered studies over the past few years lead to clear implications for (1) youth policy and (2) research practices. Additionally, while the empirical focus of this talk is on LGBQ youth, the issues and methods discussed are relevant to research on other minority groups and youth generally, and speak to survey development, methodology, and the robustness and transparency of research.

ABOUT THE SPEAKER

Dr. Joseph Cimpian is an Associate Professor of Economics and Education Policy at NYU Steinhardt and affiliated faculty at NYU Wagner and earned a Ph.D. in Economics of Education from Stanford University. His multi-disciplinary research focuses on the use and development of novel and rigorous methods to study equity and policy, particularly concerning language minorities, gender, and sexual minorities. One line of his research examines how “mischievous responders”–youth who provide extreme and untrue responses–can bias estimates of majority-minority group disparities. Some of his other work examines how beliefs about gender and math ability contribute to gender gaps in STEM. Prior to joining the faculty at NYU, Cimpian was an Associate Professor and College of Education Distinguished Scholar at the University of Illinois at Urbana-Champaign. His work has been funded by the Spencer Foundation, the AERA Grants Board, the National Science Foundation, and the Institute of Education Sciences. His research has been published in some of the top journals in education, psychology, health, and policy, and has been featured by the New York Times, USA Today, and NPR, among other outlets. He presented his work on English learner reclassification policies at a U.S. Congressional briefing and for the Council of Chief State School Officers. At NYU, he teaches intermediate and advanced graduate courses on causal inference. He is currently an Editor of Educational Evaluation and Policy Analysis, and is on the editorial boards of several other education and psychology journals.