Can a school climate survey accurately and equitably measure school quality? Examining the multilevel structure and invariance of the Georgia School Climate Scale

Abstract

School climate measures are increasingly utilized as one indicator of school quality within educational accountability systems. However, concerns have been raised about the accuracy of these indicators given that school climate surveys are often not validated using multilevel methods. Further, cross-school comparisons in climate may not be trustworthy because the school-level invariance of climate surveys has not been investigated. There is a need to examine the validity of school-level climate constructs and to determine if surveys measure climate equitably for schools that serve underrepresented populations. The aim of the current study was to examine the multilevel factor structure of a statewide school climate survey to determine whether it measured climate equitably for students of different races/ethnicities and across schools with varying racial/ethnic and socioeconomic compositions. Participants included 259,778 students from 427 middle schools throughout a southeastern U.S. state. Cross-level invariance analyses revealed that the climate constructs were measured differently across levels of analysis, and school-level climate could not be interpreted as merely the aggregate of individual-level climate. Student- and school-level factorial invariance was tested using multilevel modeling procedures. Results revealed item bias with respect to student and school characteristics, and the relationships between school climate and student and school demographics changed after accounting for identified bias. As more educational agencies consider including school climate surveys in their accountability systems, these findings suggest that multilevel validation procedures and school-level invariance analyses are necessary to ensure accurate and equitable measurement.

Keywords: Accountability; Multilevel modeling; School climate measurement; School composition; School-level invariance.