Data comparability in the teaching and learning international survey (TALIS) 2008 and 2013

This report focuses on data comparability of scale scores in the Teaching and Learning International Survey (TALIS). Valid cross-cultural comparisons of TALIS data are vital in providing input for evidence-based policy making and in promoting the equity and effectiveness of teacher policies. For this purpose, an investigation of data comparability is a prerequisite for any meaningful cross-cultural comparison. TALIS involves a large number of countries and economies, and has used rather strict conventional statistical methods to test comparability. Thus, many scales in TALIS do not reach the level of comparability that allows direct comparisons of scale scores. To facilitate the effective data analysis of TALIS and maximise its policy implications, this project: (1) uses a more flexible statistical method to test comparability, and (2) investigates the level and sources of scale data incomparability. With teacher and principal self-report data from the two rounds of TALIS (2008 and 2013), three studies are carried out to address these issues. Study 1 compares the conventional statistical method with more flexible Bayesian approximate invariance testing in scale data comparability testing. Study 2 investigates whether scale characteristics (e.g. scale length, item length, number of response options, and self-evaluative components) are associated with data comparability in principal and teacher scales. Finally, Study 3 examines the specific cultural variations that contribute to the lack of comparability. It tests the comparability of the Satisfaction with Current Work Environment scale (a key outcome construct in TALIS) between each participating country or economy with a pooled international average reference group. The paper concludes with a discussion of the implications for large-scale survey design and data analyses such as using more flexible psychometric method to test comparability and using fewer response options in items forming scales.