Maximizing impact and navigating trade-offs when scaling education innovations: A video conversation between Brad Olsen and Larry Cooley

Recently, Larry Cooley—a nonresident senior fellow in the Global Economy and Development program at the Brookings Institution and the founder and president emeritus of Management Systems International—and I sat down to talk about scaling education innovations. We found ourselves discussing two topics: (1) how to focus on the impact of the innovations, especially while adapting the scaling along the way, and (2) framing some of the trade-offs common to scaling. Our two-part video-recorded conversation focuses on these topics. Part one and part two can be seen in English here. Versions with French (part one, part two) and Spanish (part one, part two) subtitles are available.

Scaling the impact of an innovation in education is not the same as scaling the innovation.

I asked Larry how he defines scaling. He shared that the innovation (or “intervention”—we used both terms interchangeably) is just the means—not an end in itself. In public policy (unlike the private sector), it’s the outcomes of the innovation that should be scaled: Expanding or deepening the innovation is just the way to get there. Too often, however, the focus is on scaling the innovation itself—replicating or growing it so that it reaches more users. The notion of scaling for impact is different: It places emphasis squarely on the outcomes of the innovation.

For example, imagine a promising new approach to early literacy instruction that seeks to increase student literacy rates and strengthen teachers’ pedagogical practices. The innovation pilots well in three schools in a country, so it quickly expands to 200 schools. However, the program does not adequately support teachers to integrate the new approach into their overall pedagogy, and there is little evidence that it is now (after having scaled) having any effect on literacy rates in the 200 schools. Additionally, the program does not initiate corresponding changes to the surrounding student assessment system, doesn’t alter the classroom conditions in any way, and attends to school leadership only superficially in these 200 schools. The innovation may have worked as a small pilot but perhaps in ways that don’t hold true at scale.

Yet supporters can now boast of its use in 200 schools. They might report numbers of students reached or number of classrooms using the new pedagogy. They may offer evidence of teachers self-reporting that it’s a better way of teaching or surveys from students who prefer the new approach. Perhaps as a result of this apparently successful expansion, there will be plans to implement it in 2,000 schools next.

But it may be unclear whether scaling the program succeeded since there is no measurable increase in student literacy rates and the program has not yet resulted in any sustained change in teaching practice. In fact, some research suggests that many teachers revert to their prior ways of teaching literacy after a few months of an externally implemented intervention—especially after the excitement and the direct support fade away.

If the scaling focus, however, is on scaling the intended outcomes (in this example, increased literacy learning in students and improved teacher practice), the work of developing a scaling strategy, adapting it along the way, and measuring its effects will be different. Rather than scaling directly to 200 schools, a focus on impact would first work with local stakeholders to identify which aspects of the approach (and why) could be integrated into the existing education system in a sustainable way. Perhaps doing this would differ in different locations based on local needs.

For example, one location may choose to focus on providing in-depth teacher coaching on the pedagogy used in the approach to incorporate it into teachers’ broader, everyday teaching practice. Another location may choose to integrate the program content into its existing literacy curriculum and support teachers to use the new materials with fidelity. Throughout all of this, multiple kinds of data would be collected and considered to ensure that these new approaches were still contributing to the same shared goal: improved literacy for students. In some cases, but certainly not all, these varied approaches to the same goal may eventually lead to a standardized version of the intervention, applicable in a wide variety of contexts. This is scaling for impact.

Life is full of difficult trade-offs. So is scaling.

Scaling the impact of an education innovation requires painful trade-offs because—given the reality of limited time, money, and capacity—a scaling team will never get all that it wants. One common trade-off Larry discussed concerns speed versus depth. Scalers and innovation implementers are often pressured—by funding timelines, national politics, sudden windows of opportunity, and other incentive systems—to “roll out” an innovation quickly or move through the scaling process fast. But to deeply scale education outcomes—which are often multi-faceted, context-dependent, and inextricably linked to other aspects of education and communities—at least a decade is often required.

Another trade-off we raised in the video concerns cost and complexity. Many innovation developers or scalers will keep adding features to the innovation to improve its effects on children or others. Perhaps an innovation that supports socioemotional development in schoolchildren will find that addressing hunger by offering students food in the mornings improves their ability to learn, or that providing a parent-engagement dimension to the innovation will make it more holistic. That’s a noble desire, as adding useful features will indeed make the innovation’s impact better, but this will also increase its cost and complexity—two attributes that work against scaling.

We also discussed the fact that, many times, an innovation that has demonstrated success in one country is brought elsewhere to be replicated: The field calls this “education policy transfer.”Yet it’s not uncommon for an innovation that works in one situation to not work in another. As Larry comments in the video, “If you’re insensitive to the context, you’ll fail.” To address this, many innovations that are transferred from one country to another require what we call “contextualization”: the act of carefully and collaboratively fitting the core elements of the innovation into the unique cultural, policy, institutional, and human characteristics of the context. Contextualization is often essential if the innovation is going to take hold, but it requires additional time and funding—two things that are in very short supply in scaling. In both the video and a recently published report, we talk about how to strike the balance here.

Scaling is continuously learning by doing.

Our two-part conversation only scratched the surface of these and other dimensions of scaling impact in education, but we think that it offers some new perspectives on aspects of scaling that are not always discussed. After watching the videos, we  encourage you to let us know how this discussion might overlap with your own work.