Patricia Martinkova, Czech Academy of Sciences

In this work, we discuss several computational aspects of reliability estimation. We focus on two topics: We first discuss the issue of zero estimates. We then propose a flexible approach for assessing IRR in cases of heterogeneity due to covariates. The method directly models differences in variance components, uses Bayes factors to select the best performing model, implements the Bayesian model-averaging for obtaining IRR and variance component estimates accounting for model uncertainty, and it employs the inclusion Bayes factors to provide evidence for or against differences in variance components due to covariates while considering the whole model space. We focus on optimizing the estimation process in the case of higher sample sizes, higher number of covariates, and on other computational aspects connected to software implementation. Study is motivated by real data from grant proposal peer review and teacher hiring.

About the speaker

Patricia Martinkova is the vice-chair of the Department of Statistical Modelling at the Institute of Computer Science of the Czech Academy of Sciences (ICS CAS) in Prague, where she leads the Computational Psychometrics Group. She received her Ph.D. in Statistics from Charles University in 2007. She is Fulbright alumna and an affiliate at University of Washington. Her current research focuses on latent variable models, measurement error and reliability, differential item functioning, computational linguistics, and machine learning methods. She has published in journals such as Journal of the Royal Statistical Society – Series A, The R Journal, and Journal of Educational Measurement. She teaches course on statistical methods in psychometrics and hosts Seminar in Psychometrics at Charles University and ICS CAS.