Early data shows hopeful signs for pandemic learning in Philly, but huge questions remain

Early testing data shows School District of Philadelphia students have roughly held steady in math and reading achievement despite a prolonged absence from in-person learning.

The data is laden with caveats, but it’s a small indication that — in terms of academic performance — the average Philadelphia student has not regressed significantly during the pandemic.

The insights come from a research report the district published last month looking at performance on a pair of internal standardized tests that cover reading and math skills. The Star assessment was taken by students in grades 6-12 and the aimswebPlus test was administered to students in grades K-5.

Generally, these benchmark tests are used by the district to gauge whether a student is performing on grade level or lagging behind — and on which specific skills the student might need help.

This year, though — after the cancellation of statewide tests last spring — these tests are the only standardized academic metric available to judge student performance in Pennsylvania’s largest school district.

The overall picture is still troubling — most Philadelphia students are below grade level in math and reading.

But it does not appear, at least on these tests, that the pandemic has drastically altered student performance.

For example, in the fall of 2020, 24.1% of school district sixth graders were at or above grade level in reading, according to the Star exam. In the fall of 2019, that number was 23.9% — nearly identical.

A similar pattern held for students in seventh and eighth grade. Ninth and tenth-graders, meanwhile, actually scored better on the assessment this year than their counterparts did a year ago.

Math scores on the Star assessment also followed the trend. Sixth grade math scores fell slightly (about three percentage points), seventh-grade scores grew slightly (about two percentage points), and eighth-grade achievement was identical.

It was largely the same story for student scores on the aimswebPlus assessment. About the same proportion of students in fourth and fifth grade tested on grade level in fall 2020 as compared to 2019 on the reading portion of the assessment.

The lone outlier was first grade, where data showed a significant decline in reading achievement.

It’s impossible to draw conclusions from the math portion of the aimswebPlus test because so few students took the exam in fall 2019. Only this year did the district attempt to give the math portion of the exam to all students in elementary grades.

The same imbalance makes it difficult to analyze benchmark tests given to kindergarten, 2nd grade, 3rd grade, 11th grade, and 12th grade students.

Before the pandemic, the district was in the process of changing its benchmark-testing regimen across several grades, further clouding the analysis.

“It’s different for different grades because the requirements changed,” said Joy Lesnick, the school district’s deputy chief of research, evaluation, and academic partnership.

No sweeping judgments

With those uncertainties — and many others — hanging over the data, district officials say it’s too early to make sweeping statements. They also point out these tests took place in October, which means they’ve captured only a fraction of any progress or regression during the current school year.

If anything, the testing data seems to suggest that the disruptive, chaotic end to the 2019-20 school year did not throw students wildly off track academically.

Or at least that’s the case “for the grades where they are comparable numbers [of test takers],” said Tonya Wolford, the district’s chief of evaluation, research, and accountability.

Wolford also warned against taking “tiny wins” too far and said there’s still much we don’t know about how the pandemic has altered student learning.

It’s been about nine months since Philadelphia closed schools and shifted to online learning. Students in Pennsylvania’s largest school district have not returned to in-person classrooms since.

For much of that period, officials, parents, and school staff have debated the risks and rewards of reopening physical classrooms. Absent from that debate is any statistical indication of whether students are falling behind academically as they learn online.

These benchmarks exams surely do not tell the whole story. But they do fit in with other national data that shows students seem to be faring OK academically during the pandemic.

Earlier this month, a major testing nonprofit called NWEA aggregated data from 4.4 million students across the country who took the organization’s MAP Growth assessment. The organization found that students did about as well as expected in reading as they’d done before the shutdown and a bit worse in math.

“In some ways, our findings show an optimistic picture,” researchers wrote.

But the organization also noted that a significant chunk of students who’d taken the exam before the pandemic hadn’t taken a follow-up exam this fall. NWEA determined that these “missing” students were more likely from “marginalized communities,” according to NPR.

That means the data isn’t exactly apples-to-apples, and may omit the scores of students who are struggling most during the pandemic.

That same caveat may apply to Philadelphia’s data, although it’s impossible to be sure.

The district said it’s doing more granular analysis now that will look at individual student scores year over year.

“If you’re already way behind, you could be more behind [now] and these charts aren’t capturing that,” said the school district’s Lesnick. “[But] of the kids who were staying on track, they’re kind of OK.”

What tests can’t measure

Even after the district does more probing analysis, flaws in the data may remain.

Consider that this year students are taking these assessments from home.

Jeannine Payne, principal at Richard Wright School in North Philadelphia, said it took her staff considerable work to figure out how they would administer the assessments. And even then, there were some things they couldn’t control. For instance, staff had to remind parents not to help students while they did the assessment work, even while asking them to be involved in other homework.

Payne also wonders about the subset of students who have been difficult to reach during the pandemic. Even though overall participation rates in benchmark tests rose this year, it’s possible that the students who didn’t take the test are the ones struggling most in school.

“I think that the struggle we will continue to have is going to be in the comparisons,” Payne said. “We try to compare this year to last year. We try to compare these students to previous students. We’re always going to fall flat. We’re never going to have the answer.”

This hangup isn’t specific to standardized tests or the School District of Philadelphia. In general, researchers say it’s hard to know how well or how poorly school is going for students during the pandemic. Even something as basic as analyzing attendance data has taken on layers of new complications, according to Michael Gottfried, an associate professor at the University of Pennsylvania’s Graduate School of Education.

“We’re sort of building the plane as we fly it,” Gottfried said. “Everything’s up in the air now. Everything’s been disrupted just from the measurement side.”

That doesn’t mean the data is useless to administrators. For individual students, the scores give insight into what they know and what skills they’re struggling to master, said Richard Gordon IV, principal at Paul Robeson High School in West Philadelphia.

“Is it a skill we have to spend two days on or a skill we have to spend two weeks on?” Gordon said. “You can glean useful information from [the tests]. We just have to be really patient through this whole process.”

Gordon has tried to be patient with students who haven’t taken the assessments, noting that unprecedented numbers of his students are working side jobs to help their families financially.

He’s worried about the social and emotional burden this pandemic has placed on the teenagers — the kind of thing that tests cannot measure.

“They have other things on their mind that are preoccupying their time,” Gordon said.