Since schools went all-virtual in Philadelphia, students in the earliest grades are struggling to keep up academically, new data shows.
Although the data has its caveats, it presents the most complete picture yet of the academic regressions suffered by Philadelphia kids during the pandemic.
Students in grades K-5 saw larger-than-average declines on a nationally normed standardized test called aimswebPlus. The drop was largest for students moving from kindergarten to first grade — the youngest students included in the analysis.
That’s notable because Philadelphia School District administrators are pushing to bring some of those young students back into classrooms later this month, citing the academic damage wrought by online school.
It’s not all bad news, however.
A separate set of testing data shows that students in middle and high schools have tread water during the pandemic.
Taken together, this new data — analyzed and published by the School District of Philadelphia — indicates older students may get more from online learning than their younger counterparts.
“There’s a lot more to dig into, such as variation across grades, and skill levels,” said Tonya Wolford, chief of evaluation, research, and accountability. “In particular, the decline from kindergarten to first grade is notable — and aligned with the large number of foundational skills students need to learn during the early grades.”
Parsing the data
The analysis is based on benchmark tests Philadelphia students take periodically during the year.
WHYY News reported on some of the data earlier, but the analysis back then was less precise.
This data is important because it includes about 42,000 students who completed the benchmark tests in the winter of 2019-20 and again in the fall of 2020-21. In other words, it’s a way to compare before-the-pandemic scores to during-the-pandemic scores among the exact same sets of students.
Because Philadelphia was in the middle of changing its benchmark tests, this analysis only works for exams that have been around for at least two years.
For younger students, the exam that fits these criteria is the reading test called aimswebPlus. Students receive a raw score on the exam, which is then translated into a national percentile based on how other students around the country score.
In the winter of 2019-20, for instance, Philadelphia kindergarteners had an average national percentile rank (NPR) of 42.35.
The district then found 7,439 students who took the test again in the fall of their first-grade year. By that point, the students had an NPR of 27.86. That’s a 14.49 percentile rank drop.
As can be seen from the table below, that was the largest drop among all the student cohorts who took the aimswebPlus exam.
Interestingly, students moving from first to second grade suffered the smallest drop.
Philadelphia wants to bring students in second grade and below back into classrooms — a cohort that includes elementary students who have seen the largest academic dip and elementary students who’ve seen the smallest academic dip.
Even in normal years, students lose ground academically over the summer. In other words, fall test scores tend to be lower than winter test scores.
But the school district found that the drop this year was larger than normal.
Researchers ran the same analysis comparing aimswebPlus test scores from winter 2018-19 to aimswebPlus test scores from fall 2019-20.
Those students saw their test scores drop, on average, 1.87 percentage points — a drop three times smaller than the drop measured during the pandemic.
“Some of this decline is not unexpected,” district researchers wrote in an analysis. “What is striking about the decline is how large it is for students who were 2019-20 kindergarteners and 2020-21 first-grade students.”
This largely jibes with the fears of reading experts, who worried that online school could exacerbate reading struggles for young children.
It was a different story, however, for older students in grades 7-11.
Among this group, the longest-tenured exams were a pair of assessments called Star Reading and Star Math.
On these two tests, students showed basically no regression.
In a sample of about 25,000 students, the average drop in national percentile rank on the Star Reading exam was 0.09. In other words, there was little movement at all — suggesting students made expected progress from year to year.
On the Star Math exam, the average drop in NPR was 0.63 — again indicating that students have held steady during the pandemic.
Within the grade-by-grade breakdown, there is some cause for alarm. Students in grade 7 and 8 saw modest score declines in reading and math. High school students, meanwhile, seemed to actually improve academically during the pandemic.
Declines spread evenly over demographic groups
At least in terms of test scores, no one racial or socioeconomic group seems disproportionately harmed by online learning in Philadelphia.
Whether on the aimswebPlus or Star exams, students in different racial groups showed the same results year over year, on average. The same goes for students who are economically disadvantaged versus those who are not.
The charts below show average declines on the aimswebPlus exam for various racial and socioeconomic subgroups.
Are the scores legit?
These test scores are the only standardized data we have to gauge how Philadelphia kids have fared academically during the pandemic.
That makes them a useful reference point, but caveats linger.
For starters, it’s tough to gauge who isn’t being tested. Try as they might, schools will never get 100% participation on exams — and that could be doubly difficult during virtual school.
One promising sign, however, is that the number of students tested did not change much during the pandemic.
From 2018-19 to 2019-20, 41,392 students took the aimswebPlus exam. From 2019-20 to 2020-21, 41,145 students took the exam.
That suggests the data here is roughly capturing a similar set of students. It’s still possible, however, that the academic makeup of these cohorts shifted in some undetected way.
The other issue here is how the tests are being administered.
Before the pandemic, students took the tests in school and under staff supervision. Now, students take the tests at home.
There is the possibility that students are getting help from adults or giving less effort. There’s also a worry that students are simply less adept at taking tests when they’re at home and trying to navigate the exams without staff assistance.
To try to pick up on any irregularities, the district measured how long students took to complete the Star Math and Reading exams.
Researchers did find that during the pandemic students took, on average, one or two minutes longer to finish their tests. But they felt that discrepancy was “not markedly large.”
Get daily updates from WHYY News!