The President’s Suspect Statistics

This essay originally appeared at National Review online. A version of it will be published in the January 23, 2012 issue of National Review.

In early December, in Osawatomie, Kan., President Obama delivered the sort of fiery populist speech his base had been demanding since the start of his administration. The speech as a whole strongly overstated the extent of economic insecurity in today’s America, but one particular claim jumped out at me — that upward mobility has declined rather sharply:

We tell people — we tell our kids — that in this country, even if you’re born with nothing, work hard and you can get into the middle class…. And yet, over the last few decades, the rungs on the ladder of opportunity have grown farther and farther apart, and the middle class has shrunk. You know, a few years after World War II, a child who was born into poverty had a slightly better than 50-50 chance of becoming middle class as an adult. By 1980, that chance had fallen to around 40 percent. And if the trend of rising inequality over the last few decades continues, it’s estimated that a child born today will only have a one-in-three chance of making it to the middle class — 33 percent.

This claim of falling upward mobility — of diminished opportunity — rang false to me. The figures were new and of unknown origin, and they contradicted most of the research that has been conducted to date. Upward mobility is too limited in the U.S. today, and it is lower than it is in other countries (a fact cited by Rick Santorum in a recent Republican presidential debate). But upward mobility does not have to be falling for it to be too limited, and there is only the thinnest evidence that it has fallen over time. I suspected that the administration had sought out new mobility figures that would solidify the populist story of diminished opportunity that formed the basis of the Kansas speech (and perhaps of a 2012 campaign narrative).

Further research revealed that the evidence behind the president’s mobility claim is irreparably flawed. His figures are based on a very sophisticated — but unreliable — back-of-the-envelope analysis that was intended to get around data limitations. And this is far from being simply an academic question. In this case bad evidence discourages people struggling to escape poverty. It unnecessarily increases Americans’ anxiety levels and adds to the general sense of gloom that has sapped consumer confidence, thereby increasing the agonizing slowness of the recovery.

First, consider what we know from previous studies of trends in intergenerational income mobility. The bulk of the existing research shows either that mobility has increased over the long run or that it has changed little in either direction. That includes both studies that I know of examining changes in upward mobility from the bottom. It also includes six studies using measures of mobility not confined to movement up from the bottom; these find either no change or rising mobility. In contrast, only two papers find a fall in mobility, each using non-directional measures. Notably, one of them shows an uptick in the mobility of the most recent two birth cohorts it examined, leaving in doubt the question of whether the longer-term decline it found would have persisted had the authors had more recent data. The other study finds somewhat mixed evidence, depending on the data source and whether children of single parents are included.

The research to date has important limitations. Many of the studies cover a small number of cohorts, and all of the ones finding no increase in mobility use the same data set. However, the conclusion that mobility has been flat or rising over the long run is mirrored in studies of trends in mobility over adults’ lives (within a generation) and of trends in occupational mobility. The educational-mobility evidence, to my knowledge, comes from a single study that found mixed results.

The biggest shortcoming of past research is that no one has looked at the intergenerational income mobility of Americans born after the early 1970s, mostly because of limited availability of data. Children born in 1980 are barely 30 years old today, and few data sets exist that tracked them as they grew up.

However, an ongoing Labor Department survey has followed men and women born in the early 1980s, starting in 1997 when they were adolescents. In 2008, the most recent year for which their incomes are available, they were in their mid to late 20s. A predecessor of this survey (also ongoing) has followed a group born in the late 1950s and early 1960s. Using these two National Longitudinal Survey data sets, I can compare children born between 1962 and 1964 to children born between 1980 and 1982, observing their parents’ incomes when they were 14 to 16 and their own incomes twelve years later when they were 26 to 28.

In contrast to the president’s claim of declining mobility, I found that upward mobility from poverty to the middle class rose from 51 percent to 57 percent between the early-’60s cohorts and the early-’80s ones. Rather than assert that mobility has increased, I want to simply say — at this stage of my research (which is ongoing) — that it has not declined. If I include households that reported negative or no income, the rise in upward mobility I find is only from 51 percent to 53 percent, which is not a statistically meaningful increase. But the data provide absolutely no evidence that economic mobility declined, whereas the president said it had fallen by ten percentage points.

So where did the president’s numbers come from? Following up on a suggestion from New Republic writer Tim Noah’s blog that the source of the figures was Berkeley economist David Card and Council of Economic Advisors chair Alan Krueger, I reached out to Card in the days following the speech. Despite family obligations, Card graciously provided enough information for me to confirm with him that I understood how he produced the president’s figures. They turn out to come from a statistical model that we might call “Soup Cans in Six Numbers,” an elaborate attempt to estimate the “joint distribution” of parent and child incomes from aggregate income figures and various assumptions.

To understand the joint distribution of incomes, imagine standing in front of a large grocery-store display in which soup cans are laid out on a square base. At each spot on the square, a can is placed to represent both someone’s parents’ income as she was growing up, and her own family income as an adult. A can placed to the left rather than the right represents someone with low income growing up, and a can placed in the front rather than in the back represents a person who had low income herself in adulthood. If two people have roughly the same combination of childhood and adult income, their cans are stacked one atop the other.

If we created a display with 80 million cans, one for each American family, and looked to see where on the square cans piled up and where they did not, we could understand the relationship between parental income and the child’s subsequent income as an adult. Do they pile up along a diagonal ridge from front left to back right, meaning that parents and children tend to have the same income? Or is the height of the cans pretty much the same across the entire display, indicating that where children end up has little relation to where they start?

In this case, the goal of estimating the joint distribution of incomes — the shape of the soup-can display — is to determine the chance that someone growing up in poverty will make it to the middle class in adulthood. In terms of the display, the idea is to take all the cans that lie to the left of a line drawn down the display base from front to back (the line that defines “poverty” in childhood) and then within this group of cans, determine the fraction that also lie behind a line drawn across the base from left to right (the one that defines “middle class” in adulthood). The fraction of cans to the left that are also in the smaller group to the back gives the likelihood of upward mobility.

Studies typically do not have to estimate the joint distribution of incomes in this way to estimate upward-mobility rates. They have real-world data on a representative subset of families and simply do the math directly to determine how many adults were raised in poverty, and how many of them ended up in the middle class. In contrast, each of the president’s mobility estimates is based on just five numbers that describe the broad contours of the joint distribution of incomes and a sixth that defines “the middle class.”

The first four of these numbers describe inequality within the parent and child generations and average family incomes in each generation. These estimates — and a number of technical assumptions — are the basis for determining where from left to right the soup cans tend to fall, where they tend to lie from front to back, and how spread out they are in each direction. The fifth number that the modeling requires indicates the “correlation” between parent and child incomes — the extent to which the soup cans line up in a diagonal ridge or are evenly dispersed. Finally, the “median” income among adult children — the one that is right in the middle — is used to define “middle class.” (“Poverty” in childhood is defined as being below some percentile, such as being in the bottom 10 percent. Given the assumptions of the model, no real-world figure is necessary to estimate this cut point.)

One reason to doubt the reliability of the Soup Cans in Six Numbers model is the considerable work the technical assumptions do in producing its estimates. But an equally important shortcoming is the fact that the numbers on which the model is based are potentially bad proxies for the real-world numbers that are needed. Most strikingly, the level of the income correlation is simply assumed, and it is assumed not to change over time. The other figures are based on all families, young or old, with children or not, which is too broad a group. They should be confined to parents with children born within a narrow range of years and to adult children (whether they are in families or not) born around the same time. The model essentially determines how two other soup-can displays look from the front and uses that information and an assumed form for the new display to build it up from scratch. Whether the jerry-rigged joint distribution of incomes resembles the real one is an open question.

Determining how problematic this model is required that I first replicate the president’s estimates to make sure I understood the details sufficiently. When I used all the information provided by David Card, filling in a couple of gaps as best I could, I estimated that poor children born in the late 1940s had a 47 percent chance of ending up in the middle class as adults. I found that upward mobility fell, to 39 percent for poor children born in 1980 and to a projected 35 percent for poor children born in 2009. In other words, I was able to broadly replicate the figures President Obama had cited, though mine showed a somewhat smaller decline in mobility. Importantly, I discovered that the president’s estimates appear to define “upward mobility” in such a way that ending up better off than “middle class” does not count. That is, “middle class” in the president’s account means having at least a certain amount of income but no more than some maximum. My results suggest that the fraction starting out poor and ending up middle class or better is a bit higher than the president’s figures suggest, though they still show a decline from 51 to 41 percent.

I then conducted an experiment of sorts to determine what the Soup Cans in Six Numbers model would have produced for the birth cohorts I examined using the National Longitudinal Surveys. What if, instead of using the real-world survey data, I had relied only on published estimates describing average incomes for all families and the dispersion of incomes around those averages, and had assumed a specific correlation between parent and child incomes? Following the same approach used to produce the Obama figures, I found upward mobility to have fallen from 48 to 44 percent between the early 1960s and early 1980s cohorts, rather than rising from 51 percent to 57 percent, as the real-world data indicate it did.

While the president is wrong that mobility was lower for children born in 1980 than for those born previously, it remains possible that mobility for children born today will be lower. But my experiment suggests that there is little reason to think that the Soup Cans in Six Numbers model answers that question reliably. On top of all the model’s other assumptions, estimating mobility for today’s newborns requires projecting 30 years into the future the aggregate family-income statistics that are needed to describe the joint distribution of incomes.

It would be one thing if we had solid evidence that it was a lot harder to get ahead today than in the past. But in the absence of such evidence, all the president is doing is reinforcing any doubt among the poor that they can make it if they try. In the context of a molasses-slow recovery, unsupported claims of diminished opportunities also add to the forces sapping consumer confidence. The president should stick to arguing that regardless of its trajectory, upward mobility could and should be greater than it is.

Here is my preferred statistic: A poor child has less than a one-in-five chance of ending up in the top two-fifths as an adult. That’s where most readers of this essay are or will end up — would you take those odds for your own child? We have an upward-mobility problem — one that is worse than in other countries. But it is no worse than it has ever been and it does not translate into a general lack of opportunity for the middle class. That may seem like splitting hairs, but it is not. America has challenges, but diminished opportunity is not one of them.