The results of Philadelphia’s COVID-19 vaccine lottery? ‘Discouraging’

Ask us about COVID-19: What questions do you have about the coronavirus and vaccines?

Back in June, Philadelphia organized a sweepstakes program to get more people vaccinated against COVID-19. Now, the researchers who helped organize the program say it did not lead to meaningful increases in the number of people who got the shot.

Previous research about how incentives persuade people to change their behavior pointed to lotteries being an effective tool. Several states tried vaccine lotteries, most notably Ohio, which offered such prizes as $1 million and a full scholarship at a public university or college. Researchers looked at whether that worked, and found mixed results.

Linnea Gandhi, a Ph.D. student at the Wharton School at the University of Pennsylvania and lead author of the preprint research paper on the Philadelphia sweepstakes, said the goals were to get more people vaccinated and to run a more controlled experiment to see if sweepstakes would work.

“We’re at a stage in the pandemic where this research, and the mixed result coming from the state lotteries, are starting to tell a story that maybe this isn’t the best tool to use for those remaining people,” she said.

The sweepstakes research in Philadelphia was different from what other states did. For example, researchers who studied Ohio’s vaccine lottery had to compare Ohio to another state, to try to figure out what would have happened had there not been a vaccine lottery.

“Measurement is happening observationally,” Gandhi said. “It’s not a true experiment.”

In Philadelphia, the researchers could compare the vaccination rates in the city to that of other counties. But the sweepstakes program here also had a feature in which, at each drawing, the organizer would randomly draw one ZIP code from among the ones with the lowest vaccination rates, and people in that ZIP code would have a much higher chance of winning. That made it easier for researchers to isolate the effects of having a vaccine lottery. 

There were more people getting vaccinated in the run-up to the first sweepstakes drawing, but taking all the weeks into consideration, the lottery did not lead to a statistically significant change in the number of people getting vaccinated. 

The conclusion is that the lottery did not make a significant difference at this point of the pandemic, and that public officials should think about other ways to entice people to get vaccinated. 

Katy Milkman, a behavioral scientist at the University of Pennsylvania and a co-author of the research, tweeted that the results are “discouraging.”

James Garrow, communications director for the Philadelphia Department of Public Health, said, “It’s disappointing, we really wanted to be able to identify a good way for folks to be incentivized to take vaccines.”

The city had other incentive programs too — like free Eagles tickets, free Sixers tickets, free coffee at Wawa. But, Garrow said, “The city hasn’t had an incentive program that we could point to and say, ‘Look we did this thing, and it had this big jump.’”

He added, however, that the city would consider it a big deal if a program persuaded more people to get vaccinated who would not have done so otherwise.

More than 800,000 people in Philadelphia, or more than 60% of adults, have been fully vaccinated.

The findings illustrate that different groups of people respond differently to incentives, said Karen Sepucha, director of the Health Decision Sciences Center at Massachusetts General Hospital and associate professor of medicine at Harvard Medical School. She is not involved with the Philadelphia study; she is part of a group studying various vaccine incentives. That research has yet to be published, but she noted that, for example, around 34% of people in their study said they would be more willing to get vaccinated with a $100 incentive, while 15% of people said the same incentive would make them less likely to get vaccinated.

She added that the Philadelphia vaccine lottery took place in June 2021, so the people who were most eager to get the COVID-19 vaccine probably would have already gotten it.

The Kaiser Family Foundation has ongoing research on COVID-19 vaccinations, including surveys of people who have yet to be vaccinated, and those findings also distinguish between people who would definitely not get vaccinated and those who can be persuaded. Around 44% of the people in the latter group said they would be more likely to get vaccinated if the Food and Drug Administration approved one of the vaccines. That approval was granted earlier this week for the Pfizer vaccine.

Sepucha added that the fact that the study in Philadelphia was done at all, alongside actual city policy, is impressive.

“I don’t know how many meetings it took to get this all set up, but … it does speak to the commitment to science, the commitment to evidence, which is what we really need in order to understand if any of these things are working.”

Gandhi has done behavioral science experiments for companies as well, and said it’s remarkable that the city was able to run this experiment in the real world quickly.

Garrow said the city would like to keep this partnership going in the future.

WHYY is one of over 20 news organizations producing Broke in Philly, a collaborative reporting project on solutions to poverty and the city’s push towards economic justice. Follow us at @BrokeInPhilly.