The trouble with ranking cities

A few days back I took issue with a new Forbes ranking that placed L.A. near the bottom of U.S. cities when it came to job growth. There was nothing nefarious in the methodology - it's just that objective number-crunching only gets you so far in making wholesale conclusions about a region. At some point, it takes your noggin to make sense of the data, and you don't have to be Einstein to realize that L.A.'s economic prowess - even in bad times - is light years ahead of, say, Newark, N.J. or even Phoenix. As it happens, the WSJ's Carl Bialik isn't much of a rankings fan either. From his weekend column:

However rankings are designed, they can't overcome data problems. Forbes.com learned this last month when it ranked the 10 worst housing markets in the country, with Milwaukee finishing last and Denver second from the bottom. The list quickly drew scrutiny from Denver Mayor John Hickenlooper, who called Forbes to question the results. "We were in the process of coming out of [the housing downturn]," says Mr. Hickenlooper. "To hear that we're lagging just didn't make sense." Two weeks later, Forbes retracted the ranking, blaming a misuse of inventory data from real-estate site Zillow.com.

Bialik also cites a recent ranking of 140 cities worldwide by the Economist Intelligence Unit, a sister organization to the Economist magazine. Cities are rated by correspondents based on dozens of categories, and then compared with objective measures such as crime, education, and climate. Scores are weighted to give more importance to categories considered especially significant (the rankings are used by employers assigning hardship allowances). The EIU list has Vancouver as the best place to live, with a score of 98 out of 100. Vienna is second with 97.9. NY was ranked 56th, London 54th, L.A. 47th, and Detroit 40th. Calling Detroit a more livable city than London seems a bit nutty, but there are subjective considerations worth thinking about. London has a much higher chance of being struck by terrorists than Detroit, and besides, there are certain sections in the Detroit area that undoubtedly are more livable than certain sections in London. Bialik also questions how some lists are put together. From his Numbers Guy blog:

Sometimes the presentation of rankings is their downfall. A Men's Health ranking of "drunkest" American cities in February didn't include any measure of alcohol consumption, instead really measuring America's top cities for dangerous drinking, said Matt Marion, deputy editor at the magazine, who worked on the story. The ranking also counted arrests for driving under the influence against cities, though they could also reflect stepped-up enforcement of laws. "I can certainly see the other side of this," Mr. Marion said of using those arrests in that way.

All this is more than an intellectual exercise - stories about rankings are widely covered in the media (the Forbes list on L.A. job growth was all over local broadcast outlets and Web sites). Ranking stories are popular because, well, they're easy to report about and digest. That means a lot of people are drawing conclusions based on very limited bits of data.