Maria Bolsinova, Tilburg University

Adaptive learning systems (ALS) are a branch of technology-enhanced learning platforms that optimize the learning and practice material based on the student’s recent activity and provide feedback on their educational progress over time. To optimize feedback and learning material, one needs to have continuously updated, accurate, and reliable measures of the students’ changing abilities. This makes measurement one of the central issues in ALS. Measuring change is in itself not a trivial problem, however it is made even more challenging because of the adaptive nature of ALS and because they usually operate at a large scale with thousands of students needing continuous updates of their ability estimates.  These features of ALS pose challenges for traditional psychometric models and algorithms. A possible solution to these challenges is provided by the Elo Rating System (ERS) which allows to track development of student abilities and item difficulties by updating their corresponding ratings after every response using a simple formula. While this system has been broadly used in educational applications, the measurement properties of ERS have not been studied extensively. In this presentation, I will first present our results on the properties of Elo ratings: 1) the estimates are generally biased, 2) the variances are context-dependent and not known in advance, and 3) when the items are presented to students adaptively bias cannot even be defined because the ratings they drift away from each other after every response (high abilities and difficulties artificially increase, while low abilities and difficulties decrease). Second, I will introduce a modification of Elo which solves the issue of ratings drifting away from each other. Third, I will present a new urn-based rating system called Urnings which solves not only the drift issue, but also the problem of bias and unknown variance.

About the Speaker

Maria Bolsinova is an assistant professor at Tilburg University, the Netherlands. Maria obtained her Ph.D. from Utrecht University in 2016 after which she worked as a postdoctoral researcher at the University of Amsterdam and a research scientist at ACTNext. Her dissertation entitled ‘Simple models and complex reality: Contributions to item response theory in educational measurement’ has been recognized by the dissertation awards from the Psychometric Society and from the National Council on Measurement in Education. Maria has also received the New Assessment Research Award from the Association of Educational Assessment Europe. Her research has been devoted to developing advanced psychometric models for the analysis of product and process data from educational tests and to building statistical foundations for measurement in adaptive learning systems.