Assessing BIDs Using Propensity Score Matching

By Reid Ewing

Those of you who read this column regularly (thank you!) may know that I have written only one other highly technical column, on spatial econometrics. My apologies in advance for another highly technical piece, on propensity score matching.

Past columns have dealt with quasi-experimental research. The most common quasi-experimental design has one group receiving treatment (treatment group), and another group, matched to the first, receiving no treatment (control group).

The "treatment" can be anything from a zoning change to a street closure (Research You Can Use, April 2010). If we compare the two groups after the treatment, we can equate the effect of the treatment to the difference in some relevant outcome between the two groups. The control group provides a so-called counterfactual, showing what might have happened in the absence of the treatment. We cannot observe a counterfactual in the treatment group since the group in question has, by definition, been treated. But by observing the control group, we can at least intuit what might have happened in the absence of the treatment.

All of this assumes, of course, that the groups are comparable to begin with.

The Journal of the American Planning Association is publishing (it is currently online) a quasi-experimental study that compares travel patterns of households in two corridors in Portland, Oregon. My colleague Shima Hamidi and I were the coauthors. In this case, light rail was added to one corridor and the other remained strictly highway-oriented.

To see if household travel had changed in the corridor with light rail, we compared household vehicle miles traveled for the two groups in 1994, four years before the opening of the transit line, and in 2011, 13 years after. Household VMT was comparable for the two groups before light rail, but very different after. The light-rail line had made a positive difference, reducing VMT per household.

One of the reviewers of this study advocated the use of propensity score matching in lieu of the matched corridor study design we chose. Propensity score matching is just another way of creating counterfactuals. We resisted, and the study was published anyway. But it got me thinking about propensity score matching as a promising way of conducting quasi-experimental research.

Propensity score matching is fairly new to planning. A Google Scholar search using keywords "propensity score matching" and "urban planning" produced only 185 hits, many of which were not actual applications. One actual application will be published in an upcoming issue of the Journal of Planning Education and Research (it is currently online). Stacey Sutton of Columbia University has applied the technique to the assessment of business improvement districts in New York City.

During the 1970s a handful of cities, starting with Toronto, then New Orleans and New York City, adopted the BID model as "a novel mechanism for financing and managing the revitalization of forlorn commercial corridors and downtown shopping districts," Sutton's study says.

By 2009 more than 1,500 BIDs had been established in hundreds of urban and suburban municipalities across the U.S. and internationally. BIDs proliferated in New York under Mayor Michael Bloomberg, who promoted them.

BIDs provide a variety of place-based services — such as street cleaning, security, streetscape enhancement, and facade improvement — to enhance overall commercial life. BID operating expenses are largely covered by a compulsory tax assessment on businesses. Since the self-financing mechanism of BIDs becomes a tax on retail tenants, Sutton's study examined whether the BID model improves retail performance relative to comparable areas of the city that never adopted BIDs. She is one of the first to systematically examine the utility of the BID model.

How to select "comparable areas" for purposes of such an assessment? In the New York case, not all BIDs have obvious comparables, and that is where propensity score matching comes in. Sutton matched BIDs to non-BID census tracts based on observed pre-BID attributes known to affect BID adoption (retail density, assessed property value, and many other variables). This exercise produced a credible control group of non-BID census tracts that have a high probability of BID adoption but have not done so.

The most common matching approach, known as nearest neighbor matching, matches each case in the treated group to the case in the control group that has the most similar propensity score. In Sutton's case, each BID tract was matched with two "nearest neighbor" non-BID tracts with the closest propensity scores to create a matched trio.

Sutton's primary finding is that both sales and employment declined for existing independent neighborhood retailers within BIDs relative to comparable non-BID areas. She notes that BID functions and capacities are highly variable and tend to be correlated with BID size. Commercial vibrancy observable in large corporate BIDs such as the Times Square BID may not be obtainable by much smaller, community BIDs.

The use of propensity score matching reduces potential bias in selecting treatment and control cases, here by building those cases "up from the data." The use of propensity score matching leads to a different conclusion, in Sutton's paper, than a simpler research design might have. That is a good thing, as truth is the ultimate goal of research.

Reid Ewing is a professor of city and metropolitan planning at the University of Utah and an associate editor of the Journal of the American Planning Association.