“Apples to outcomes?” Revisiting the achievement v. attainment differences in school voucher studies

The summer of 2022 has seen a flurry of legal and policy efforts to expand publicly funded private school choice programs. These include: the U.S. Supreme Court decision in Carson v. Makin, which ruled that voucher programs cannot exclude religious schools; Arizona’s creation of a near-universal voucher program; and various state-level actions, such as a ballot initiative led by Betsy DeVos to create a tax credit-based voucher program in my home state of Michigan.

I, and many others, have studied voucher programs for a number of years now. After more than two decades of research on vouchers, a general pattern has emerged in the results.

The effects of voucher programs on student attainment (how long students persist in school/college) appear at least somewhat positive while the effects on student achievement (what students know as measured by standardized tests) appear very negative. Voucher advocates also point to positive impacts on survey measures such as school satisfaction or safety as well, but studies employing these outcomes vary in quality and method and are not generally included as a major evaluation focus.

It’s also important to understand the timing of this research. The early voucher studies—roughly 1998 to 2005—focused on programs that were generally city-based and relatively small (e.g., in Milwaukee) and found some positive effects on student achievement. This is in stark contrast to more recent studies that have shown clearly negative effects on student test scores in places where vouchers have expanded into large programs, such as Louisiana, Indiana, Ohio, and Washington, D.C. Some of these negative effects have been extremely disconcerting.

The effects from Louisiana, for example, approach 0.5 standard deviations in math—more than double some estimates of even the COVID-19 pandemic’s effects on learning loss. These initially harmful test score effects often seem to persist over time, too.

On the other hand, a few of these studies have shown improvements on outcomes such as high school graduation or enrollment in college (e.g., in Milwaukee and Washington, D.C.).

This leaves us with a bit of a puzzle. How can we reconcile findings that voucher programs lead students to perform worse academically with research suggesting they might also lead students to persist longer in school?

As a researcher on vouchers since 2005, I’ll offer a few possible explanations.

First, I believe the positive effects on attainment are often overstated. In Louisiana, for example, the study that found negative effects on test scores found no significant effects on college enrollment. And apart from the genuinely large effects of the small program in Washington, D.C., the positive effects on attainment that are apparent in the literature are very modest in size (especially compared to the shockingly significant negative effects on test scores). This includes our initial findings from Milwaukee.

Notably, too, our slightly positive findings on college enrollment in Milwaukee are partly a story of students from Catholic high schools entering Catholic colleges, which raises questions of whether these gains might come more from networking than increased educational productivity. In fact, the studies that track students for the longest periods of time—from New York City and Milwaukee—find that any attainment effects fade away as students work their way through college. There’s little evidence of voucher impacts on rates of college persistence or graduation. As a case in point, a follow-up paper on D.C. vouchers found no effects on college attainment advantage among those who had initially shown a higher chance of graduating high school.

Second, for the most part, the seemingly inconsistent effects on attainment and achievement come from different students, schools, and/or places. That is to say this literature generally does not come from a single group of students who experienced both negative effects on their test scores and a boost to their persistence in school. In some cases, this is because the attainment and achievement studies come from altogether different cities or states. In other cases, it’s because they come from different students attending different schools within the same site. For example, in Milwaukee, we studied achievement effects primarily in grades 3-8 and attainment with a separate cohort of 9th graders (with limited follow-up for students in earlier grades). This leaves us to speculate whether the positive attainment results are driven by some private high schools that are succeeding in ways that their private primary school counterparts are not.

Third, it’s worth noting that private schools—especially if not subject to rigorous state guidelines or oversight (as most are not)—might be able to improve graduate rates by simply lowering their standards and making it easier to graduate (in ways not available to public schools). Some writers have argued that public secondary schools prefer graduation as an accountability measure because those rates are easier to manipulate than test scores. If that is true, the same logic applies at least equally, and probably more so, to private schools that are marketing themselves to would-be consumers. This story would also help explain why voucher students are generally no more likely to persist in college.

This potential tradeoff between attainment and achievement, if it even exists, isn’t just an academic argument. Parents have to make decisions about which schools their children attend—maybe more than ever if private school choice programs continue to expand. And we have some data on what parents actually value. We know from voucher application data in Washington, D.C. and New Orleans (that is, based on the characteristics of the schools they actually choose, not what they say in a survey) that academic quality is the dominant determinant of private school choice. Other factors like distance from home to school, safety, religious education, or after school programs matter too, but private school parents still appear to overwhelmingly prioritize the academic success of their child.

All of this leads me to suggest that it’s time to reframe the question that has been guiding the debate on voucher effects. That question has been, “Which outcome matters more, achievement or attainment?,” with some voucher advocates arguing forcefully against paying much attention to test scores. Instead, maybe we should be asking, “What would it take to offset the dismal learning loss induced by vouchers?”

Joshua Cowen is a Professor of Education Policy in the College of Education at Michigan State University. He also currently serves as a research advisor to the new federally mandated evaluation of the D.C. Opportunity Scholarship (voucher) Program.