MASSACHUSETTS RESIDENTS WILL soon decide whether the state should continue using the Massachusetts Comprehensive Assessment System (MCAS) as a graduation requirement for public high school students, something it has done since 2003. A “yes” vote on Question 2 would remove the graduation requirement, but not the test itself nor the state standards that guide teaching in the Commonwealth. A “no” vote would leave the graduation requirement in place. Proponents of a “no” vote often make the following arguments in support of their position:
MCAS scores predict future educational attainment and even future earnings.
Student achievement increased starting in 2003 once the graduation requirement first went into effect.
Without the MCAS graduation requirement, we are giving up on students.
We now have enough evidence to fact check these claims and, just as importantly, the analysis comes from a trusted and perhaps unlikely source: the very report cited regularly as evidence for the points above.
“Lifting All Boats?” from the Annenberg Institute at Brown University remains the most extensive analysis of the relationship between MCAS scores and student outcomes, especially college enrollment, college graduation, and average earnings at age 30. As a testing skeptic, I expected that I’d disagree with the key arguments of the report as well as a more recent companion that focuses on the impact of the graduation requirement. Instead, I found my perspective further confirmed by key aspects of the research that testing proponents simply leave out. Most notably:
MCAS scores don’t “predict” future educational outcomes and earnings, at least not in the common sense definition of that term.
When there is a high correlation between two variables, it is standard for a statistical report to say that one factor (e.g., high MCAS scores) “predicts” the other (e.g., high earnings). Outside of statistical reports, this statement might imply causality. But, the authors of “Lifting All Boats?” are straightforward in rejecting such claims, noting: “We cannot make causal claims about the relationships we present. For example, while we document that students with higher MCAS scores have better earnings, we cannot conclude that raising MCAS scores definitively causes higher earnings.” Indeed, decades of educational research suggest that family income is among the most influential factors driving both test score outcomes and future earnings.
To be fair, the report gestures towards causality when it groups students according to 8th grade MCAS scores and demographics, including family income. In this analysis, students who improve their MCAS scores between 8th grade and 10th grade go on to enjoy higher earnings than peers with the same demographic profile and 8th grade scores whose MCAS performance decreased in 10th grade. This suggests that learning experiences in high school are important drivers of future success, an argument that presumably has universal agreement among supporters and opponents of Question 2.
It’s critically important, though, to remember that the ballot question will not remove the MCAS entirely, only the graduation requirement. And, it is on this point that the Anneberg study again offers evidence for a “yes” vote.
The authors are clear that “educational attainment” — or high school and college graduation — is a unique factor driving student earnings over and above their 10th grade MCAS scores. They argue that “closing the existing gaps in high school performance and postsecondary educational attainments could dramatically reduce current levels of income inequality.” Of course, retaining the MCAS graduation requirement means that we would retain a major barrier to postsecondary educational attainment. From the very data in this study, we know that students from low-income families represent a disproportionate share of those denied diplomas — and subsequent postsecondary opportunities — under the current system.
For even a supporter of the MCAS as an assessment of academic knowledge, then, a “yes” vote aligns more closely with the recommendations of the report, because it allows room for both recommendations to take place in Massachusetts schools. School leaders will still be able to use the test to measure high school performance while the graduation requirement will not be a barrier to educational attainment.
The critical question, then, is whether the high-stakes nature of the graduation requirement actually helps to lift all boats. Is the graduation requirement itself the spark that drives improvement for all students? Again, the evidence says otherwise. Indeed:
Student achievement increased starting in 2003, but then plateaued just 5 years later.
Just like the argument above, the strength of this one relies on an assumption of causality that does not exist in the research evidence. On this point as well, the Brown University study is clear. It does not identify the graduation requirement as the reason the scores increased from 2003-2008. Importantly, the report also notes that any increases since 2008 are basically a mirage: “increases in average MCAS mathematics and ELA scores until approximately 2008 represented real gains in academic proficiency, while the increases since then have been almost entirely due to scale drift, score inflation, or a combination of the two.”
Also just like the argument above, money — in this case school funding — hides in the background as the most likely causal factor. Of course, the same law that created MCAS also provided school districts with dramatically increased state funding. Decades of educational research have likewise proven that increased funding drives improvements in student achievement. (For this very reason, the Student Opportunity Act is essential for the future of Massachusetts public education, though we’ve sadly seen the bill’s funding increases blunted by unprecedented levels of inflation.)
So, as evident in the favored report of MCAS supporters, the current system provides an advantage for wealthy students, and for at least the last 16 years, it has provided no direct causal benefit for all students.
Nonetheless, proponents of the graduation requirement frame it as an equity issue, arguing that without it, we are giving up on students. Again, data from the same research project highlight the exact students who are harmed the most in the current system:
With the MCAS graduation requirement, we may well be giving up on English learners, especially newcomers to the United States.
Of the students who never pass MCAS, even after retakes, 85 percent are English learners or students with disabilities. Nearly 1 out of every 4 are newcomers to the United States. And, these numbers are based on older data that do not account for important recent developments: newcomer arrivals have skyrocketed in recent years and the state is struggling to meet their needs. Meanwhile, we’ve raised the benchmark for a passing score on the ELA MCAS. This is a perfect storm for increased failure rates and dropouts.
Newcomers face challenges that others simply do not have to face in learning academic content in their non-native language. The average voter might not know that they are expected to take the MCAS just one year after arriving in the United States. This, to me, indicates that we have an incompetent measuring stick, not incompetent or unskilled students. Again, the Annenberg research supports this argument.
There is one important caveat in the findings noted above: For English learners specifically, the correlation — or “predictive” value — between MCAS scores and future earnings completely disappears. This is likely because “MCAS scores for students who were English learners understated their latent academic skills.” In more simple language: MCAS scores just aren’t a good measure of what English learners know because they are taking it in a new language with their very educational future on the line.
Nonetheless, the current system tells them that they can’t graduate if they don’t pass MCAS, even if they are passing their coursework and even if their language proficiency is improving as measured by another test called ACCESS. In fact, these are all viable components of a graduation requirement that might replace the MCAS. Without a change, we are offering these students a dead end.
Because legislatures often need urgency in order to take action, a “yes” vote would likely further accelerate our state conversation about a potential new graduation requirement. Meanwhile, a “no” vote simply leaves in place a system developed over 30 years ago, when our state looked very different and when we knew less about the relationship between MCAS and student outcomes.
Our students, in particular our English learners, deserve better from our educational system. And, the voters — who will decide this policy — deserve better than the careless repetition of unsubstantiated claims from supporters of the status quo.
Peter Piazza is a research assistant professor at the University of Massachusetts Amherst and school quality measures project director at the Massachusetts Consortium for Innovative Education Assessment and Education Commonwealth Project.
The post Report often cited by MCAS backers isn’t as pro-MCAS as they say appeared first on CommonWealth Beacon.