The promise of integrated data systems for social policy reform: A Q&A with Dennis Culhane and John Fantuzzo, principal investigators, Actionable Intelligence for Social Policy

Actionable Intelligence for Social Policy (AISP) is an initiative funded by the John D. and Catherine T. MacArthur Foundation, with the purpose of improving the quality of education, and health and human service agencies’ policies and practices through the use of integrated data systems.  AISP aims to build a nationwide professional network of data expertise and to help executive leaders in municipal, county and state governments use integrated data systems to better develop and evaluate their programs. As we have noted in recent papers, well designed and integrated data systems are a key to determining how to improve economic mobility and health in neighborhoods.

Dennis Culhane is the Dana and Andrew Stone Professor of Social Policy at the School of Social Policy and Practice at The University of Pennsylvania, and John Fantuzzo is the Albert M. Greenfield Professor of Human Relations at the Graduate School of Education at The University of Pennsylvania.  

What problem led you to start the Actionable Intelligence for Social Policy Center? 

Public agency analysts as well as their collaborators in academic research frequently want access to administrative data from a variety of public agencies to track complex, multisystem populations and to use public program data to answer important policy and practice questions. They also want multiple data sources to examine the transitions of subpopulations across the life course, such as youth as they enter the workforce.  In most places, getting access to multiple databases will take at least a year, and sometimes two years or more to negotiate.  Starting over with each request adds unnecessary delays, and often kills any interest and data analytic possibilities.  

Integrated data systems solve that by creating an infrastructure both for the ongoing storage of linkable data and for the processing of requests, making it much easier and more feasible to execute research, evaluation, and planning projects in a timely manner.  That timeliness turns out to be crucial in that the results of the analysis tend to be more actionable by the participating agencies, and this generates further support and enthusiasm for the work, thereby increasing the sustainability of that infrastructure.  

We formed AISP as a peer-to-peer professional network for states and counties and their academic partners who are using integrated data systems to tackle complex social problems.  We have studied their best practices and their common limitations.  We have also developed multisite research projects to demonstrate their capacity to address important social problems.  And we have been working to educate other states and counties, as well as federal agencies and foundations, about the importance and potential of this approach for improved evidence-based policymaking.

How does an integrated data system work? Who are the key actors involved in supplying, receiving, and centralizing the data?  

Integrated data systems store agency data from multiple departments, such as health (e.g. vital statistics, disease registries, Medicaid paid claims), human services (child protective services, homelessness), housing (public housing and vouchers), education, (K-12, early childhood, post-secondary), workforce (training and vocational rehabilitation), earnings, and justice programs (juvenile justice, jails and prisons).  The system is governed by representatives from the data-providing agencies who oversee the policies and procedures, who approve of proposed projects, and who review results.  A dedicated staff is responsible for managing the data flow, for processing requests, and for data security.

In the AISP 2014 Project Brief, you suggest that by pooling their disparate databases, social service agencies can better address the often interconnected needs of those they serve, more efficiently and at less cost. Why are integrated data systems so valuable in enabling government agencies to do this? 

Virtually every agency spends disproportionate resources on a relatively small number of people or families, often with complex needs.  These same people are also often users of multiple systems.  So, one primary, common interest among agencies is to identify these subpopulations and to figure out how they can be better and more efficiently served.   And because of their heavy service use, the potential for cost savings is also greatest among these groups.  Agencies can also use these data to inform potential interventions, and, ultimately, to test these interventions in real time, while watching the impact on agency program utilization and related outcomes.  Recent enthusiasm in improved evidence-based policymaking has also piqued interest in using linked administrative data to enable higher speed and lower cost randomized controlled trials to test social program innovations.

On your website, you identify 13 networks of organizations that have set up functioning integrated data systems capable of producing actionable intelligence to guide policy. These include the states of Michigan and South Carolina, and the County of Los Angeles. What caused these governments to take the step? What have been some of the measurable outcomes? 

Different jurisdictions get interested in building integrated data systems for different reasons.  Often there is a galvanizing issue of interest to the mayor or governor, like school readiness, that brings people to the table initially.  Other places have simply tired of the cumbersome and lengthy process of executing data-sharing agreements every time there is a need.  And in some cases, an agency executive has a vision for the potential for integrated data and puts a system into place. 
Success can be measured by the many, many projects done by these different sites.  These projects have been used to improve programs, create efficiencies, and improve lives.   

You distinguish four broad sets of challenges facing those who develop, implement and use such systems. Could you elaborate on these? Is there a specific challenge, or a group of challenges, that are the most difficult to overcome? 

We have in our recent book, Actionable Intelligence, laid out a developmental hierarchy of four cumulative and progressive steps to developing, implementing, and using an integrated data system (IDS) to continuously improve government public services: legal agreements to share data, state-of-the art scientific methods to guarantee data quality and computer science technology to protect data security, governance processes to ensure ethical uses, and attention to political and economic sustainability. 

At each step there are distinctive challenges that must be resolved to establish a robust system. A common challenge to sites looking to develop an IDS is overcoming the perception that data sharing for policy research and planning is not permitted due to legal barriers.  However, it has been demonstrated by the many sites building these systems that nearly all the legal barriers can be addressed as sites clearly indicate the value of research to improving services.  But even experienced sites still face challenges in getting data-use agreements negotiated with external researchers and university attorneys, since the benefits to government agencies is not as clear as the benefits to the independent researchers. 

Concerns with data quality and security are also common, but physical, technical, and training standards can overcome those.  

Establishing cross-agency governance processes is of utmost important to the integrity and internal consistency and equity of uses of IDS. Having an entity appointed by the executive leader of the state or municipality to oversee use and to ensure that the government’s priorities are the primary focus of use is essential. It is important to have a forum to address the needs and ethical issues of all participating agencies in routine use of the IDS. Lastly, well-developed sites are always focused on the sustainability of their IDS across changes in political leadership and fluctuations in the economy.  As such, these sites are constantly looking to develop sustainable funding sources and to make sure that they develop diverse means of communicating to their leadership and the public that their services substantially contribute to improving services and cutting costs.  Government-based sites are usually better and more securely funded, whereas universities often have to seek grants to stay operational. However, university-based sites are less influenced by the ebb and flow of the internal political agendas of executive administrations. 

Over the next few years, what are some of the objectives you are hoping to accomplish with Actionable Intelligence for Social Policy? What lessons or recommendations do you have for states and local governments hoping to build an integrated data infrastructure? 

In addition to studying best practices of effective IDS sites in state and local government, we have also been studying what are the ceilings/limitations of our best IDS sites. Studying limitations to state-of-the-art models has surfaced targets for innovations; these are unique areas that must be addressed with new and creative ideas to permit IDS to meet the increasing needs for the use of integrated administrative data to provide a means of evidence-based decision making. 
In our model of IDS, we have depicted the configuration of key contributors to the effectiveness of these systems as a diamond. At the apex of the diamond is executive leadership with the authority and appropriations to provide mandated public services. Along the vertical axis of the diamond is the public, which both creates the auspices for the elected leadership and is the recipient of the services. The horizontal points of the diamond are those who professionally implement and evaluate the services – the practitioners – and those who transform the data into information to inform decision making-researchers. 

We have through our study identified specific needs for innovation pertinent to each of these key contributors and their transactions with each other. We are partnering with national funders to advance prototypes to raise the ceilings that currently are in place at each point of the diamond.  We have prioritized the apex for obvious reasons and we are poised to start a new initiative to address the primary ceilings of IDS use from the executive leadership’s perspective; these include data security, versatile and cost-effective technology, and time-efficient and low-cost transactions in the use process. This initiative, AISP Innovation, is designed to create policy and practice standards for the field of integrated data systems.  We are going to convene experts in legal issues, research ethics, data security, and data science to develop guidelines that reflect the best practices in the field, and that take into account the common federal and state laws.  These standards will then be used to design several technology and procedural solutions that we can offer at low- to no-cost to states and counties interested in launching an IDS, thereby reducing implementation barriers, as well as time and costs.  

Our next innovation focus will address the ceilings of use for practitioners and researchers.  Here innovations will focus on developing the human resource capacity for wider and more effective use of IDS by training government practitioners, analysts, and academic researchers in the most effective approaches to the development and use of IDS.  And we are always searching for opportunities to demonstrate the value of these systems through multi-site research projects that focus on important social issues like early childhood education, disconnected youth, homelessness, and prisoner re-entry.  Creating communities of learning across states and local governments through visible multi-site projects will generate more exemplars of use and more opportunities for the public to see the benefits of IDS use over perceived risks.  It will be essential for expanded IDS use to establish a more transparent dialogue with the public through innovate and transactional communications.

In conclusion, AISP is playing a vital role in helping to build the infrastructure and expertise for using integrated data systems in order to analyze social programs and foster policy innovation.  As we have found in our recent work, there are many difficulties involved in sharing data.  Problems include agency silos as well as inadequate capacity and skills within organizations.  These hamper the evaluation of promising reforms and initiatives, and frustrate innovative organizations and jurisdictions.  But by helping jurisdictions with such assistance as designing data use agreements, solving legal obstacles, and developing effective governance policies, AISP is significantly improving the capacity for governments to improve social policy and encourage creativity.