Pennsylvania ditches school scoring, but will parents?

A couple years ago, Jean Rauscher was at a familiar crossroads.

Her oldest child was about to start school and the family wanted to move.

“And we can’t visit everywhere,” she said. “So thank god for the internet.”

Rauscher’s first instinct was to open up a web browser and google the phrase “best elementary schools in the USA.” It led her to a suite of websites: GreatSchools, Niche.com, SchoolDigger, and US News and World Report. Each is a little different, but they all try to judge school quality. And they each attach an overall score, letter grade, or ranking to schools across the country.

“It’s so helpful to have that one number,” she said. “This elementary school’s a 10. This elementary school’s an 8.”

Schools in the Tredyffrin-Easttown district in suburban Philadelphia rate highly, and that’s eventually where the family moved. The ratings didn’t dictate Rauscher’s choice, but they guided it. They gave her somewhere to start.

Pennsylvania used to be one of the entities that rated schools.

In 2013, the state started assigning School Performance Profile scores, which ranged from 0 to more than 100. That kind of scoring was once required by the federal government. Then the law changed, and Governor Tom Wolf’s administration says it started to rethink its approach.

Last month it unveiled its new school evaluation tool, the Future Ready PA Index.

If you visit the site you’ll see a profile page for each of the commonwealth’s public and charter schools. Those profiles contain a collage of dots sorted into eight categories — with links to more detailed data.

But here’s what you won’t find:

You won’t find a number. You won’t find a letter grade. You won’t find any sort of bold-faced metric to tell you whether the school is “good” or “bad” or somewhere in between.

Pennsylvania officials believe this “dashboard” approach deemphasizes standardized test scores and offers parents a better, more nuanced look at public schools across the state. Ideally, they hope it will change the behavior of schools and districts, freeing them to focus on more holistic measures of student success.

Skeptics wonder, though, if the state’s new tool stands a chance against established, attractive, and privately run sites that millions of people already trust.

Will parents like Jean Rauscher take time to scroll through a detailed dashboard, when a simple Google search quickly produces intuitive snapshots they can use to map their kids’ futures?

An age of ratings

In some ways, Pennsylvania officials are swimming against the cultural current.

We live in an age of ratings. Yelp, TripAdvisor, Rotten Tomatoes — each provide numerical guideposts, giving consumers a seemingly rational way to sift through the bewildering array of options now at their fingertips.

Education is not immune to this trend.

But some researchers believe a one-score rating system for schools is fundamentally flawed.

Since getting the green light from the federal government, eleven states have scrapped their rating systems in favor of descriptive models that assign words like “average” or “needs improvement.”

Another five, including Pennsylvania and California, have gone a step further. They no longer provide any sort of top-line or “summative” score, meaning they make no definitive declaration about the quality of their schools.

“A single summative score can send a false message about what’s happening throughout a school,” said Matthew Stem, the state’s deputy secretary for elementary and secondary education.

Stem and his colleagues also worry the old model relied too heavily on test scores, which they see as incomplete at best, and misleading at worst.

“[Test scores] mask what’s happening under the surface,” Stem said.

Other critics go further.

They say ratings don’t tell you how good the teachers are or how good the curriculum is. Instead, they tell you whether a school has a lot of wealthy students, kids who are more likely to score well on tests due to factors beyond the school walls. Those high test scores become high ratings, which then attract more wealthy people who have the income to afford homes and taxes in highly-rated districts.

“You can see a sort of feedback loop here,” said Jack Schneider, an assistant professor of education at the University of Massachusetts at Lowell who studies school-quality measures. “What we can see is that information is leading people to self-segregate.”

High wealth, high scores

The correlation between high-wealth schools and high school ratings is clear.

The graphic below uses state data from 2017 and shows schools that scored well on Pennsylvania’s School Performance Profiles tended to be wealthier. Schools that served more poor students fared markedly worse.

That same phenomenon shows up in the 10-point-scale rating system used by GreatSchools, a nonprofit that has become one of the most prominent independent school-ranking services.

Analyzing data provided by GreatSchools, Keystone Crossroads found that, again, wealthy schools tended to score better.

On average, Pennsylvania schools where low-income students made up a quarter of the student body or less scored a 7.5.

Pennsylvania schools were low-income students made up three-quarters of the student body or more received, on average, a 2.7 rating.

Even if this correlation didn’t exist, some see a philosophical problem with reducing schools to a score.

“Schools don’t lend themselves to this sort of rating,” said Schneider. “Schools are highly complex. They’re more like ecosystems than hotel rooms.”

Simply by existing, a school rating endorses the idea that schools can be uniformly “good” or “bad.” The truth, Schneider said, is more complicated. He believes parents should evaluate whether a school fits their child, but the current paradigm encourages them to sort schools first into a hierarchy before they do that kind of deeper analysis. And while the idea of “good” and “bad” schools predates the internet, at least one recent study suggests the proliferation of school ratings has exacerbated wealth segregation.

Pennsylvania’s new evaluation tool is designed to change the way parents and community members look at schools.

Each school’s home page features eight categories, and they receive a blue, green, or red mark in each. Blue is the best and red is the worst. The state also indicates if it believes a school’s performance in each category is on the rise, stable, or declining.

Four of the eight categories reference standardized test scores on state exams, with two looking at overall proficiency and two looking at student growth. There are also categories dedicated to English language learners, attendance, and college and career readiness.

‘That one number’

The question is whether this kind of nuanced format will catch on.

Some state lawmakers have already been questioning whether the new approach makes sense.

Earlier this year, John Eichelberger, R-Blair, the former head of the Senate Education Committee, introduced a bill that would have legally codified School Performance Profiles as the state’s sole accountability measure. As Eichelberger sees it, the new dashboard model will only confuse people, and he worries that the lack of summative score will effectively allow schools to dodge criticism.

“We have 500 school districts in Pennsylvania — they’re some of the best public relations machines that I’ve ever seen,” said Eichelberger. “We hear all this good news coming from schools.”

When people hear their local schools aren’t scoring well on SPP or other metrics, they’re often surprised, he said. He sees the Future Ready PA Index as a back-channel to avoid the scrutiny that comes with hard data.

“The education bureaucracy is never satisfied with any of the numbers they get,” he said. “They have a difficult time being assessed.”

The toughest people to win over, though, will probably be parents, like Jean Rauscher.

Rauscher admits her family might have focused more on ratings than others, but lots of families do some version of the search she and her family conducted and find it very useful.

As Kenneth Hong, another recent Tredyffrin-Easttown transplant put it, rating systems like GreatSchools and Niche “gave us a high comfort level that [the district] has pretty high quality schools.”

Hong and Rauscher acknowledge that ratings sites are probably biased toward schools in wealthy districts with hyper-involved parents. But every type of evaluation has its imperfections, including word of mouth.

Besides, Rauscher said, the bias baked into the rating systems could be a good thing.

“That’s gonna be your community, people who want their kids to learn, and do well, and go to really good colleges,” she said. “I don’t have a problem with that.”

‘A complex problem’

GreatSchools received 47 million unique visitors last year, the organization said, helped by its link with real estate sites such as Zillow. About 22 million people visited the K-12 ratings on Niche.com, one million of which logged on in Pennsylvania, according to a spokesperson.

GreatSchools’ Chief Strategy Officer Sam Olivieri acknowledges that the relationship between school ratings and student poverty creates a “complex problem.” But she argues that simply getting rid of summative scores won’t improve things.

Parents need some sort of top-line score before they can really roll up their sleeves and start looking at schools, she says. It’s the portal that brings them to all the other information GreatSchools has on its site — data on class size, absenteeism, and demographics.

“It’s a starting point,” Olivieri said of the overall score.

The key is to improve ratings systems, she believes, not bury them.

In 2017, GreatSchools changed its formula to include an “equity” rating for each school, which looks at how historically disadvantaged groups score on state tests and contributes to a school’s overall score.

GreatSchools also added nuance to its rating system by factoring in student growth on state tests and not relying solely on overall proficiency. The formula also takes into account graduation rates and advanced offerings for high schools.

Based on a Keystone Crossroads analysis, these changes have slightly diminished the strong correlation between student wealth and school rating.

Can nuance survive?

Even some skeptics of school-rating sites caution against abandoning the model. Information overload, they argue, can confuse parents, turning a potentially empowering resource into something that further separates haves from have notes.

Rating systems that lack a “summary measure are not really helpful,” said Rebecca Jacobsen, an associate professor for education politics and policy at Michigan State University. “Most parents will simply turn off and not really understand the data.”

Jack Schneider isn’t a believer in summative scores, but he thinks Pennsylvania’s new site needs “a design intervention, stat.”

“I do this for a living and spent maybe 15 minutes on it. And struggled for a while to figure out what I was looking at,” he said.

Matthew Stem, with the state’s department of education, said his colleagues focus-tested the Future Ready PA Index with parents and so far the feedback has been “almost universally positive.”

But this is all new, and the question remains: Can states offer a more nuanced tool to gauge school quality without alienating the people they’re supposed to help?

As Olivieri from GreatSchools’ warns:

“If you build it, they won’t necessarily come.”