Personalized learning has grown increasingly popular in recent years, encouraged in part by a 2015 RAND study showing large gains from the practice. But newer research has found more modest achievement gains and identified implementation challenges. At first glance, this might seem like bad news, but a more encouraging picture emerges by looking at details of the two studies together.
In the earlier study, my co-authors and I provided evidence from the largest, most-rigorous evaluation of student achievement in schools adopting personalized learning across all grades and subjects. These schools seek to break away from age-based classrooms—where all students learn the same material on the same day—in favor of providing an individualized educational experience to every student. The study found large positive achievement gains in mathematics and reading. Students started below national norms for their age and grew to surpass those norms two years later. These results helped to set high expectations for personalized learning as the practice spreads.
Last month, our new study reported smaller achievement gains for personalized learning schools. Results were still positive in both mathematics and reading, but the achievement gains were modest, and statistically significant only in mathematics.
Interpreting results across studies
How can educators, funders, and policymakers make sense of the differing results from the two studies?
First, it is important to consider the differences in the sets of schools included. The 2015 sample included only schools that had been implementing whole-school personalized learning for at least two years, and many were members of charter networks that were early pioneers of the concept; more than two-thirds of students were in elementary grades. In contrast, nearly half of the 2017 sample were in their first year of implementation, and only a few were members of those pioneer charter networks; three-quarters of the students were in secondary grades.
Those sample differences likely contributed to the differing results in the two studies. Education research studies often find that the largest effects occur in the younger grades. (The reason is unclear, but may simply be a result of how achievement is measured and analyzed.) It is also common for innovations to take time to gain traction and sometimes even produce a dip in achievement the first year implemented, before maturing to show more positive effects. Both of these facts favor the 2015 sample and could help explain larger effects in that group.
Research also commonly finds that when innovations are spread to new contexts and scaled up to serve larger samples, they produce smaller effects than were seen among the earliest adopters. This, too, favors the 2015 sample and probably explains some part of the differences in effects measured in the two studies.
This problem of replication and scale-up speaks more directly to the question of whether personalized learning holds promise to dramatically improve educational outcomes for all students. If effects diminish when implemented more broadly, it will not have the magnitude of impact that many are hoping to see.
A more optimistic interpretation? Personalized learning continues to look promising because positive effects were evident in the 2017 study despite these hurdles. Of course, work remains to validate this sign of promise using the most rigorous experimental research methods. Meanwhile, the 2017 results could help other sites implementing personalized learning set realistic expectations: The size of effects will likely depend on the grade levels involved, may take several years to fully develop and may not fully replicate the results seen for the earliest adopters.
Learning from success and challenges
The new study also shows huge variation in the effects estimated for individual schools, with some seeming to replicate the large positive effects seen for early adopters and others producing strong negative effects. Although the study cannot explain results school-by-school, it does offer many details about how personalized learning is implemented, noting some factors that may hinder the realization of its full potential.
One example: Teachers reported concerns related to personalized learning’s goal of identifying each student’s level of achievement, focusing on material the students are ready to learn, and giving them time to master that material before moving on. Theoretically, these practices can be beneficial for all, and particularly for students who have fallen behind, because they will work on attainable goals, achieve the gratification of success, and become equipped for more advanced work. We have not yet seen how this strategy will play out over the long term, but it has the potential to lead to greater achievement. Think of “tortoise” students, who develop strong fundamentals they can capitalize on later, versus “hare” students, whose schools have them racing through content underprepared and thus aren’t given the chance to develop the deep understanding needed for long-term success.
However, many teachers in the new study reported discomfort with the slow pace of some students, and such concerns can be magnified by existing policies around grade-level standards and assessments. Students who start well below grade level could make extraordinary learning gains, yet not adequately cover the material on end-of-year state tests. In some cases, teacher performance may be judged by student performance on those tests. These tensions can be further aggravated by another aspect of personalization–giving students choices in the content and sequence of their studies–because the material students learn might not align well to the standards and assessments. Personalized learning might not be able to flourish until these tensions are relieved by policy adaptations around standards and accountability.
Another challenge identified by the study: There are too few ready-made resources to draw on to provide more personalized pathways and activities for students. Teachers reported that this puts additional burdens on them to develop personalized content, taking time that might be better spent working directly with students, or forces them to limit the amount of customization available to students. In the short term, educators must consider this burden on local staff, while in the long run funders and curriculum developers must seek to provide more materials that can be readily assembled into customized learning plans.
The new evidence continues to suggest that personalized learning holds promise as an innovation that can lead to improved educational outcomes for students. But implementers should have modest expectations for the magnitude of the benefits, and patience for the full benefits to emerge. It is also important to consider the challenges of implementing this innovation and the policies and resources that may be necessary for it to prosper. Meanwhile, it will be important to advance this “evidence of promise” through rigorous causal studies and to develop a greater understanding of the critical components of this complex innovation.
Related Content
