Estimating the Effectiveness of City Connects on Middle School Outcomes

Abstract

City Connects is a school-based model that identifies the strengths and needs of every student and links each child to a tailored set of intervention, prevention, and enrichment services in the school or community. The purpose of this study was to conduct a comprehensive evaluation of the City Connects treatment effects on academic performance (both MCAS scores and grade point average (GPA) grades) in middle school using student longitudinal records. Parallel analyses were conducted: one evaluated the City Connects elementary intervention (serving kindergarten to fifth grades) and the other one evaluated the City Connects middle school intervention (serving sixth to eighth grades). A series of two-level hierarchical linear models with middle school achievement scores adjusted and/or propensity score weights applied were used to answer the research questions of interest. In addition, to make a causal inference, a sensitivity analysis was conducted to examine whether or not the estimated treatment effects resulted from the first two analyses were robust to the presence of unobserved selection bias. The results showed that students who were exposed to the City Connects elementary intervention significantly outperformed their counterparts, who graduated from the comparison elementary schools, on academic achievement in all middle school grades. However, in the case of the City Connects intervention schools that served middle school grades, since all students only received a maximum of one year of City Connects middle school intervention, it was still too soon to expect any significant changes. Moreover, the estimated treatment effects of the City Connects elementary intervention were only mildly sensitive to the presence of some forms of hidden bias, which made the causal inference of City Connects on middle school academic achievement quite plausible.