City Controller Alan Butkovitz held a press conference today to criticize Mayor Michael Nutter’s communications with businesses in advance of the papal visit.

Citing a survey of hospitality businesses located within the traffic-restricted “Francis Festival Grounds” conducted by his office, Butkovitz reported that the top three concerns were getting employees to work, securing enough supplies and deliveries, and the removal of trash. The security perimeter will prohibit automobile traffic, preventing deliveries and trash pickup services over the weekend.

Only 68 of the 250 businesses contacted responded to the survey, a 27 percent response rate. The survey letters were delivered on August 18th and businesses were required to complete the survey by August 28th. The city’s Papal Visit Business Resource Center, which aims to help impacted businesses deal with logistics during the papal visit, opened on August 17th.

When asked about the response rate, Butkovitz’s director of financial and policy analysis Jeff Hornstein handled the response: “We feel like it was a pretty comprehensive list, and getting 68 responses [out of 250 businesses contacted] is better than statistically significant.”

But when asked earlier whether these 68 businesses were a representative sample, Controller Butkovitz responded: “It wasn’t designed as a statistically—it’s not like a presidential poll.”

Without indication of whether the respondents were representative of the impacted businesses as a whole (30 restaurants, 9 retail establishments, 27 hotels and 2 “other” responded), or an argument to support treating the responses of disparate industries as statistically interchangeable, there seems to be little reason to claim this is statistically significant.

But the intricacies of statistical significance are lost on most.

If one uses a simple formula for calculating statistical significant response rates, you would need a sample size of 152 out of a population of 250 to achieve a 95% confidence level with margin of error of +/-5 percent and a response distribution of 50 percent (which is admittedly high; but 10 percent would still require 90 responses).

Unscientific survey responses are often treated identically to statistically significant polling.  Even when reporters are careful to make that distinction, percentages still impart the imprimatur of objectivity: “29 percent” sounds quantitative, whereas “a handful” makes it clear that we’re getting merely qualitative responses at best. The City Controller did the later: he collected a handful of responses.

This was a press conference to announce a few dozen anonymous anecdotes.

And Butkovitz admitted as much. When challenged on the statistics, Butkotvitz said that the purpose of the survey was to communicate the logistical challenges retail businesses in the Francis Festival Grounds to the administration.

Anecdotal evidence can be great, but they aren’t statistics. 

Most of the businesses that chose to respond to the survey said they received insufficient security details to plan for the weekend, and faced significant logistical issues due to the traffic conditions. These are legitimate concerns, if not legitimate statistics.

But politicians and partisans frequently provide nonsense numbers like this because it makes their points sound more authoritative, and too often the media simply parrots the nonsense back, allowing the most damning number dominate the top line.

And the top line numbers that Controller Butkowitz clearly wants to see in every lede were those he first mentioned, the figures he placed on a placard next to the podium. That pie chart is below. 

This isn’t to say that the papal visit communication strategy has been stellar. It hasn’t, and the recent change in tone from the Sturm und Drang pronouncements early on to “I’ll be there,” may be too little, too late.

But there is reason to believe that the papal panic may be just that: panic, the kind of irrational fear that spreads like a virus. Butkovitz said that the change in responsiveness and tone “appear to have calmed some of the businesses’ anxiety.”

But if Butkovitz was sincere in that statement, and sincere that he was merely trying to ensure that the business community’s concerns and needs were addressed, it isn’t clear how publically publishing this survey, which will likely only further anxiety among area businesses who may now wonder why they haven’t been too concerned, accomplishes that.