Manqian Liao, Duolingo

Date & Time: Thursday, July 22 at 9:00AM EST

Computational psychometrics, a blend of theory-driven psychometrics and data-driven algorithms, provides the theoretical underpinnings for the new generation of data-rich assessments. High-stakes digital-first assessments are assessments that can be taken anytime and anywhere in the world and their scores impact test takers’ lives. The unprecedented flexibility, complexity, and high-stakes nature of these digital-first assessments poses enormous quality assurance challenges. In order to ensure these assessments meet both “the contest and the measurement” requirements of high-stakes tests (Holland, 1994), it is necessary to conduct continuous pattern monitoring and be able to promptly react when needed. In this paper, we illustrate the development of a quality assurance system, Analytics for Quality Assurance in Assessment (AQuAA), for a high-stakes and digital-first assessment. To build the system, educational data from continuous administrations of the assessments are mined, modeled and monitored. In particular, five categories of statistics were monitored to assure the quality of the assessment, including scores, test taker profiles, repeaters, item analysis and item exposure. Various control charts and models were applied to detect and alert for the abnormal changes in the assessment statistics. The monitoring results and alerts were communicated with the stakeholders via an interactive dashboard. The paper is concluded with a discussion on how the automatic quality assurance system is combined with the human review process in real-world application.

About the Speaker

Manqian (Mancy) Liao is a psychometrician at Duolingo. Currently, her work focuses on conducting validity research on the Duolingo English Test, including developing and improving the quality assurance process of the test and conducting differential item functioning studies. She got her Ph.D. degree in Educational Measurement, Statistics and Evaluation from the University of Maryland, College Park in 2020. Her research interests include item response theory and cognitive diagnostic modeling. She is passionate about utilizing her expertise to find innovative solutions to improve validity and fairness of the test scores under the applied settings.