Philly district suspends school rating system, seeks fix

The Philadelphia School District revealed Wednesday that its system for rating schools is faulty

The district has suspended use of its so-called “School Performance Index,” or SPI.

District leaders are now seeking outside help to fix the complicated formula that converts a dozen factors into a single score given to every public school in the city, including charters.

For the past two years, SPI scores have been used to help guide a wide range of major decisions, including which schools should be closed down or converted into charters. It has also been used to evaluate charters’ bids for renewal or expansion.

Leaders of several of the city’s charter schools have long taken issue with the index.

“We are at this point confident that there were some mistakes made,” said new district Deputy Superintendent Paul Kihn. “We honestly don’t know how extensive the problem is.”

Kihn said the issues with SPI stem from human error in how the accountability measure was calculated – not from faulty data resulting from cheating on state standardized tests.

“They are two different matters,” said Kihn.

A state-commissioned analysis of results from 2009-11 found evidence of widespread cheating at dozens of schools across the state, including 53 traditional public schools and three charters in Philadelphia. Last year, after tight new test security measures were put in place, scores at many of those schools dropped dramatically – in some cases by 50 percentage points or more.

“I am not prepared to comment about the cheating matter in relation to this,” said Kihn.

A far-reaching tool

The district developed its School Performance Index in 2009, when Arlene Ackerman was superintendent. The formula boils down either 13 indicators (for elementary and middle schools) or 17 indicators (for high schools) into a single score of 1-10, with 1 being the best.

For all schools, indicators relating to student scores on the Pennsylvania System of School Assessment (PSSA) exams are weighted most heavily, and measures of parent, student and teacher satisfaction also taken into consideration. High schools’ success in preparing students for college and careers also factors in.

In an interview Wednesday, Kihn acknowledged the central role that SPI has played in the district’s Renaissance Schools initiative, facilities master planning process, and decisions regarding charter schools. But Kihn stressed that the rankings were never “used as the sole determinant for any decision that we have made.”

The measure will be disregarded altogether as district officials target several dozen more schools for possible closure by next fall, he said:

“Going forward, we’ve made the decision that we will not use SPI at all in the facilities master planning process.”

Transition a factor

The district first became aware of a potential problem with its performance index last May, said Kihn. In response, officials conducted an internal, three month-long investigation that concluded in late August. Action was not taken earlier because of the district’s recent change in leadership, he said:

“Because [incoming Superintendent William Hite] and I were just transitioning in, a decision was made to hold on taking any action until we could be briefed and make the decision about how to proceed.”

The district’s office of accountability remains responsible for developing and maintaining the School Performance Index.

In August, the School Reform Commission approved the termination of Daniel Piotrowski, then the district’s executive director of accountability and assessment, effective July 14.

Neither district officials nor Piotrowski would comment on whether his termination was connected to the problems with SPI.

Earlier this month, NewsWorks and the Philadelphia Public School Notebook reported that Piotrowski, who also served as the head of the district’s testing security program, had called in March for a full investigation of the testing procedures at Wagner Middle School, where he had reported more than a dozen testing violations. Piotrowksi was removed from Wagner and other officials overturned his judgment that an investigation was warranted.

Charter concerns

Some charter operators have criticized SPI since its inception, claiming the district’s numbers were not trustworthy and criticizing the formula.

“I’m glad that they recognized [the problem],” said David Hardy, the CEO of Boys Latin charter in West Philadelphia.

But Hardy, who has long advocated for an accountability measure that puts more emphasis on students’ postsecondary outcomes and less on their test scores, said his issues with the performance index go beyond a “math problem” resulting from faulty calculations.

“What they need to do is change the types of things they’re measuring and the weight of their measurements and take a broader view of success,” said Hardy. “In an environment of school choice, parents have to have [good] information so they know what they’re buying.”

Kihn acknowledged such concerns and said he personally notified leaders in the charter community of the problems with SPI on Wednesday morning.

Kihn also said he recently received a proposal for how to improve the performance index from a working group of the Great Schools Compact committee, which has convened district officials and charter leaders to examine the issue for months.

“We are certainly taking those very seriously as a set of thoughtful recommendations,” said Kihn.

Most immediately, though, the district is now looking for an outside vendor who can recalculate SPI scores from 2009-10 and 2010-11, as well as to calculate 2011-12 scores for the first time.

Kihn said he hopes to receive bids by mid-November and that the work will take no more than a few months. All results and analysis of SPI in the future will be publicly posted to encourage transparency, he stressed.

Making recommendations about how to improve the measure will be optional for whichever vendor is selected, said Kihn.

This story was reported as part of a partnership in education coverage between WHYY/Newsworks and the Public School Notebook.