In late October, the Maryland coronavirus testing positivity rate as calculated by Johns Hopkins University dropped significantly, but not because the state’s percentage of positive cases declined.
Instead, the COVID Tracking Project, a site from The Atlantic online news magazine devoted to collecting nationwide COVID-19 information, changed how it tracks Maryland’s data to include all viral tests performed.
Hopkins previously had reported Maryland’s seven-day positivity rate as the number of new cases divided by the number of new people who tested in a weeklong span. That method was different from the state’s, which divided new positive test results by all test results over the course of a week.
The COVID Tracking Project’s change means both rates now have the same denominator while keeping their original numerators.
Dr. Jennifer Nuzzo, lead epidemiologist for Hopkins' Testing Insights Initiative, said Hopkins made no changes to how it calculates positivity.
“Our formula has not changed, to be clear,” Nuzzo said. “It’s pulling from the same data categories on COVID Tracking’s website. However, COVID Tracking changed what they put into that data category, so for states, we’re using the category ‘total tests,’ which in the case of Maryland, had actually previously reflected persons tested.”
Maryland’s and Hopkins' rates are far closer now. For data through Tuesday, Maryland had its seven-day rate as 3.2%, compared to Hopkins' 2.36%. Using Hopkins' original methodology, its rate for Maryland in the same timeframe would be 5.98%.
Throughout the pandemic, states have varied in how they report coronavirus data. For example, some states will count one individual who is tested twice as two individuals as long as that pair of tests came more than a week apart, Nuzzo said.
As the pandemic has gone on, states have changed the data they make available. For example, Maryland originally referred to its count of residents who had tested negative as the actual number of negative test results. As states have continued to publicize and clarify data, both Hopkins and the COVID Tracking Project have worked to create a semblance of nationwide consistency.
“In the early days of the project, our only way of estimating total tests completed by states was to add the positive and negative tests together,” said Alexis Madrigal of the COVID Tracking Project. “But as time has gone on, more and more states now provide time series of total tests. This is better data. So, as we are able to get a hold of this data for different states, we switch our default total tests to those numbers.”
The greater issue, Nuzzo said, is that there are no federal data standards requiring states to all issue the same data in the same way, which would allow accurate comparisons. Nuzzo said she and others are working to encourage those who can create those standards to do so, but for now, the inconsistent data leads to the same phrase, “positivity rate,” meaning different things.
Hopkins uses the COVID Tracking Project’s data to compare the rates of all 50 states, but some of the rates measure the percentage of people who test positive, some measure the percentage of tests that come back positive, and some — like Maryland’s now — measure the percentage of tests that were performed on first-time test recipients who indeed had COVID-19.
In a conversation with The Baltimore Sun, Nuzzo detailed the challenges and confusion those inconsistencies create and what the next steps could be. Questions and responses were edited for brevity and clarity.
What caused the rate to change?
We have generally preferred to take a people-centered view of positivity. What happened with the ‘rewiring’ of the data into the categories was that what got put into the denominator are tests that we previously didn’t include in our calculation, and it puts the calculation on our website now closer to what Maryland presents as its own calculation, which is a test-centered positivity calculation. Both of them are accurate, and both of them are important to look at, but we would like to see calculations that also look at the people who are being tested and what percentage of them are coming back positive.
The challenge that we have now because of this data change and because of these ongoing issues where there are no federal data standards and states are reporting data in very different ways, it’s making it basically impossible for us to do a single calculation for all states. We may need to look at positivity calculations in multiple ways and not just one. But we’re very much trying to dig into the data and see what’s possible, given the fact that we don’t have any [federal] standards and not every state reports the same type of data ... You know, 10 months into this pandemic, it would be nice if we had some consistency in the way these data are reported.
Hopkins compares the seven-day positivity rates for all 50 states, plus Washington, D.C., and Puerto Rico. Before this change, Maryland’s rate was in the middle of the pack. Now, it’s among the 10 lowest. Are all of those rates measuring the same thing?
No, they aren’t, and that has been the case from the beginning because there aren’t data standards. The way states report data, they are reporting apples and oranges. ... States have made more data available. However, they have done so in the absence of any kind of standards that ensure that they are all calling what basically is the same word the same thing. When some states report number of people tested, places like Maryland deduplicate at the person level. If you’ve ever been tested before, you don’t show up in that number. Other states deduplicate on the week, so as long as you were not retested this week, you will show up again the next time you’re tested. They sound exactly like the same category, but what states choose to put into that category differs because there are no federal data standards.
COVID Tracking [Project], I think, has been somewhat heroic in their efforts to try to track down the data that states are providing and to try to provide some interpretation to it, but they of course are dealing with all of these changes. ... I think from our perspective, the uncertainty is increasing as a result of these changes, and for this reason, I think it’s becoming increasingly clear that we are going to have to start calculating positivity not just in one way, but in multiple ways.
Is Hopkins' current Maryland rate of new cases divided by new tests, a people-over-tests calculation, an acceptable method for measuring positivity?
It’s not inaccurate. Actually, the CDC put out a definition of how you can calculate positivity, and they articulated three different ways, and it was people over people, people over tests, or tests over tests. States are doing all three in their own calculations. I think our preference is, where possible, to have an apple over an apple versus an apple over an orange. But the reality is when we’re dealing with messy data in the real world, we have to do the best that we can with the best data that we can find, and those might not be the ideal data. If we had federal data standards, it would be a lot easier.
What are the potential impacts of states' rates being calculated differently?
This really underscores that states should really be wary when they make high-consequence decisions based on other states' positivity data because if they don’t fully understand the context behind the numbers and what goes into that state’s calculation, they may be making interpretations that are not quite right. …
We know some states use positivity as the sole metric for deciding what states to put on its quarantine list. Even if we had an ideal positivity calculation where we were absolutely confident that we’re comparing apples to apples between states, I don’t think that that is an appropriate way to use positivity... Aside from that sort of misunderstanding of what positivity actually is, it’s really not a good idea to do when states are reporting data in different ways.”
The people tested number, Hopkins' original denominator for Maryland’s rate, is still listed on the COVID Tracking Project’s site. Is Hopkins going to change its mapping and pull that figure again? Is positivity going to become more of a state-by-state calculation? Basically, what happens next?
“One strong possibility is a state-by-state approach based on the data that are available. I still think it’s important for us to be tracking people tested, so I imagine if we are able to do that for a state, then we would want to do that, but also recognize it’s also important to see the impact of these screening programs that are being implemented in nursing homes and in colleges, so maybe, percentage of tests coming back positive is also important.
"It’s becoming increasingly hard to use the same calculation for all states, and yet, there still is value in putting out a visualization of what is happening across the United States rather than only having 50 separate visualizations.”
Baltimore Sun reporter Phil Davis contributed to this article.