This blog is a summary of the new report “Toward data-driven education systems: Insights into using information to measure results and manage change,” which is available to download here.
When data advocates promote evidence-based decision-making in education systems, they rarely specify who the intended users are, for what purpose, and what kinds of data are needed. The implicit assumption is: by everyone, for everything, and any data.
But since collecting, processing, and communicating data require substantial resources, it is prudent to assess whether data produced are indeed accessible and valuable to key decision-makers. Surprisingly little systematic research exists on the types of information education decision-makers in developing countries value most—and why.
In a new report, Toward data-driven education systems: Insights into using information to measure results and manage change, the Center for Universal Education at Brookings and AidData offer insights to those very questions. We analyze the results of two unique surveys that asked education policymakers in low- and middle-income countries about their use of data in decision-making. Survey participants included senior- and mid-level government officials, in-country staff of development partner organizations, and domestic civil society leaders, among others. (For more details on the surveys, see page 18 in the report.)
The report aims to help the global education community take stock of what information decision-makers actually use and offer practical recommendations to help those who fund and produce education data to be more responsive to what decision-makers want and need. We summarize the findings below:
Finding 1: Having enough information is seldom the decisive factor in making most education decisions; instead, decision-makers desire to have sufficient government capacity.
Enacting education policies, changing programs, and allocating resources are complex decisions that demand weighing multiple factors, such as having sufficient capacity and financial resources, having enough information, and having the support of the public. So where do data and information fall within a decision-maker’s cost-benefit analysis?
We found that information is not as important as technical capacity, financing, and political support. Some decisions, however, depend more on having sufficient data and information, such as creating or abolishing schools or grades, and testing students. One possible explanation could be that leaders feel they need strong justification (via an evidence base) for these decisions which could become easily politicized.
Finding 2: Education decision-makers use evidence to support the policymaking process, for both retrospective assessment and forward-looking activities
But while information may not be the most decisive factor in education decisions, its role is significant. We found that decision-makers in the education sector are more likely to use data and analysis as compared to other sectors (such as health and governance), including for forward-looking purposes, such as design and implementation of policies or programs, as well as retrospective assessments of past performance. As shown in Figure 2, most education sector decision-makers (over 70 percent) report using data or analysis fairly consistently throughout the policymaking process.
Finding 3: Education decision-makers most often use national statistics from domestic sources and program evaluation data from international sources.
Decision-makers overwhelmingly rely on national statistics from domestic sources and program evaluation data from international organizations. The high use of national statistics points to the salience of such data for each country, including, for example, dropout rates for primary school students by district or municipality, the number of schools providing secondary education in each village, or pupil-teacher ratios in urban vs. rural areas.
Finding 4: Education decision-makers consider administrative data and program evaluations most essential, and want more of the latter, signaling a gap between need and supply.
We asked leaders about their wish list—what types of information would they want more of? We found that those who allocate and manage resources place a premium on administrative data (e.g., number of schools, teachers, students) and government budget and expenditure data (e.g., school-level budgets, expenditure per student). Meanwhile, those working on personnel management need teacher performance data, whereas leaders tasked with overseeing instructional matters need program evaluation data and student-level assessment data. Given respondents’ wish lists, we identified four opportunities for data producers to respond to unmet demand: (1) program performance and evaluation data; (2) budget and expenditure data; (3) student-level assessment data; and (4) teacher performance data.
Finding 5: Education decision-makers value domestic data that reflect local context and point to policy actions, and improving the timeliness and accessibility of information will make it more helpful.
Having identified some of the gaps that exist in meeting the needs of education decision-makers, we asked what producers and funders of data should do better or differently to meet the data demands. Leaders said that data from both domestic and international sources were most helpful when they provide information that reflects the local context. They also viewed information from international sources as most helpful because it provides policy recommendations (43 percent) and is often accompanied by critical financial, material, or technical support (36 percent). Leaders viewed domestic data as helpful when it was available at the right level of aggregation, as well as timely, trustworthy, and insightful.
When asked what improvements producers could undertake to make data more valuable, respondents suggest improving the timeliness and accessibility, as well as improving data disaggregation, accuracy, and trustworthiness. The respondents requested data from the national government, in particular, to be more accessible and disaggregated.
Finding 6: Decision-makers strongly support strengthening their countries’ education management information system (EMIS) to bolster their education data ecosystem.
Beyond finding general areas of improvement for education data, we also asked respondents to rank a list of specific solutions. Respondents largely agreed on the seven solutions proposed, rating all of them as “extremely important”, on average. But of the seven solutions, the recommendation to strengthen the EMIS within the education ministry resonated with the highest number of respondents.
Moving from data generation to impact
The path from data generation to impact is not simple, automatic, or quick. The seemingly straightforward story of information supply, demand, and use is complicated by users’ norms (how they prefer to make decisions), relationships (whom they know and trust), and capacities (their confidence and ability to turn data into actionable insights). The process of moving from data generation to use and, ultimately, to impact on education outcomes must also take into account the different institutional environments (i.e., political context) that may incentivize or dampen efforts to make decisions based upon evidence.
Most essentially, though, investments in data creation must be matched by an equal (or greater) emphasis on increasing the use of evidence by decision-makers, built from a strong understanding of what data and information they use, value, and want. Understanding why education decision-makers and influencers do not notice, value, or use data that are produced by their own statistical agencies or by international organizations deserves more attention than it has received thus far.
Authors
Commentary
6 key insights into the data and information education leaders want most
February 20, 2018