Making Sense of Different Results: Is Meta-Regression Necessarily Best?

By Reid Ewing

The Winter issue of JAPA contains what may be the first application of metaregression to the planning field: "Does Compact Development Make People Drive Less?" by Professor Mark Stevens of the University of British Columbia. Meta-regression is a new approach to meta-analysis, something I have written about in this column previously (March 2007 and March 2009). A meta-analysis is a study of studies that uses individual studies as data points to estimate the size of relationships among variables. It is powerful in that it generalizes across an entire literature. Think of it as a literature review in objective, quantitative terms.

Until meta-regression, meta-analyses simply averaged results across studies, weighting studies appropriately. Now, with meta-regression, we can model relationships in much the same way we regressed dependent variables on independent variables in our first statistics class.

Several months ago, I started getting emails from Professor Stevens asking for clarification about methodology, results, and conclusions in the 2010 JAPA article I coauthored with Robert Cervero, "Travel and the Built Environment: A Meta-Analysis." Shortly thereafter, I received a request from JAPA to review a paper by Stevens that challenged our findings and conclusions.

Stevens's article has been published in the Winter issue of JAPA, as has a detailed counterpoint by Cervero and myself, which I summarize in this column.

There is much to commend in Stevens's article. It is methodical, novel, and clearly written. I have no doubt that it will be widely cited and used by practitioners. But it also raises serious concerns:

Overreaching Conclusions. By far my greatest concern is an overreaching on conclusions. Stevens's main message is "compact development has limited potential for making people drive less." This is all the casual reader will take away from his article. But his actual results, as well as mine and Cervero's, tell a more nuanced story. Let's not give up on planning for compact development just yet.

For example, Stevens describes the elasticity of vehicle miles traveled per capita with respect to population density, at –0.22, as "small." Compared to what? In reality, an elasticity (which, as a refresher, is a measure of one variable's sensitivity to a change in another variable) of –0.22 means that a doubling of density (100 percent increase) results in a 22 percent reduction in driving. Imagine the typical city if 22 percent of cars suddenly disappeared. That's hardly small.

Benefits and Costs. My next concern relates to the way costs and benefits are depicted. From Stevens's article, you might conclude that the sole benefit of compact development is a reduction in driving.

I cannot begin, in this short column, to document all the benefits of compact development beyond that of reduced driving. Researchers have found benefits in increased walking and transit use, reduced residential energy consumption, reduced pedestrian and motor vehicle fatalities, increased physical activity and reduced obesity, reduced household transportation costs, increased upward social and economic mobility, and increased social interaction and neighborliness.

Stevens's article may also cause readers to despair for the high political costs of compact development. He focuses exclusively on one D-variable (density) and correctly states that single-family neighborhoods are generally opposed to densification in their backyards. In Salt Lake City, where I live, there has been opposition from single-family neighborhoods to high-density development nearby, particularly to transit-oriented development. What has not generated opposition is high density at a distance, for example, downtown or in commercial centers, or land-use diversity, good street design, and the other D-variables nearby.

Sample Size Requirements. Finally, Stevens states, "Meta-regression analysis, which I use to produce my findings, is the most objective and statistically rigorous approach to systematic reviews of a body of literature currently available." This might be true, if Stevens's sample size were not inadequate for meta-regression. It consists of only 37 studies, and as few as 18 individual analyses. Readers will recall from their basic statistics course the rule of thumb that multiple regression requires a minimum sample of 50 cases total or 10 cases per independent variable. In our original article, Cervero and I had an issue with sample size too, but we simply reported descriptive statistics rather than applying inferential statistics via metaregression analysis.

ELASTICITIES OF VMT WITH RESPECT TO D VARIABLES: Findings from two JAPA-published studies that sought to determine how compact development impacts driving rates show how different samples and different methods can lead to very different results. The takeaway for planners: Methodology matters. Sources: R. Ewing and R. Cervero, "Travel and The Built Environment: A Meta-Analysis," JAPA, 2010; M. Stevens, "Does Compact Development Make People Drive Less?" JAPA, 2016

Trying to control for biases

None of this would be particularly problematic if the new study did not lead to different results than our older one. The graph on the next page reports Cervero's and my original weighted average elasticities, the weighted average elasticities for Stevens's larger sample of studies, and Stevens's estimated elasticities after attempting to control for two confounding influences using meta-regression — so-called self-selection bias and publication bias. For a discussion of these biases, and why Stevens cannot necessarily control for them, see our counterpoint article.

In some cases, you can see the elasticity values in the three analyses are very similar. For example, elasticities of VMT with respect to distance to transit are virtually identical. In such cases, it probably does not matter which values planners use. In other cases, results are very different. For these, planners have a distinct choice. Does land-use diversity (mixed uses) reduce driving? We say it does. He says it doesn't. Readers, take your pick.

Many academic planners, to avoid the appearance of bias, throw up their hands when 90 percent of the evidence points in one direction and 10 percent in the other. I believe Stevens has fallen into this trap and that practitioners who rely on academics' research deserve better.

Reid Ewing is chair of the Department of City and Metropolitan Planning at the University of Utah, an associate editor of the Journal of the American Planning Association, and an editorial board member of the Journal of Planning Education and Research and Landscape and Urban Planning.