Modeling Item-Level Heterogeneous Treatment Effects: A Tutorial in R

Half day short course (9:30am – 12:30pm)

Short course #3

Program evaluators typically examine total test scores as outcomes of educational interventions. However, a recent body of scholarship has demonstrated that interventions can have unique impacts on each item of an outcome measure. These models for “Item-Level Heterogeneous Treatment Effects” (IL-HTE) provide both statistical and substantive insights into the impacts of educational interventions and have implications for critical issues in educational research such as learning transfer, alignment, score inflation, and fade out. In this training session, we will review the logic behind IL-HTE models and learn how to estimate them in R using real data from randomized controlled trials in education research. Knowledge of item response theory, multilevel models, and causal inference in randomized trials is assumed. Attendees should bring laptops with R and RStudio installed. Prior experience with R is helpful but not necessary.

Intended Audience

Researchers interested in the intersection of psychometrics, causal inference, and program evaluation.

References

Gilbert, J. B. (2024). Modeling item-level heterogeneous treatment effects: A tutorial with the glmer function from the lme4 package in R. Behavior Research Methods, 56(5), 5055-5067. https://link.springer.com/article/10.3758/s13428-023-02245-8

About the instructor

Joshua Gilbert (Harvard University Graduate School of Education)

 Joshua Gilbert, PhD