Back in September the pro-Affordable Care Act (ACA) website Vox touted a new working paper by Jonathan Gruber and Robin McKnight. There was nothing explicitly dishonest in the study or the coverage of it, but Vox’s readers would walk away with a very misleading impression of what Gruber and McKnight had actually discovered. In this post I’ll illustrate the problem with an analogy to travel, where the rhetorical sleight of hand will be obvious.
Consider the Vox article titled, “Limited doctor choice plans don’t mean worse care.” As we’ll see, the Gruber/McKnight study doesn’t actually show that at all. Let me quote from the Vox article to set the context:
The idea of narrow networks — health insurance plans that limit enrollees to a small set of doctors — is not a concept that’s especially popular with consumers. Who wants a health insurance plan, after all, that tells you your favorite doctor is one it won’t cover?
Narrow networks are common on Obamacare‘s new exchanges, as health insurers try to hold down premium prices by contracting with fewer doctors….This unsurprisingly led to a barrage of negative headlines about insurers leaving well-known hospitals out of network.
But narrow networks aren’t all bad. Health economists actually tend to be quite fond of these products, as they help hold down spending. The potential for savings is big: limited choice plans can reduce patient spending by as much as a third, new research from economists Jon Gruber and Robin McKnight finds.
OK, so the Gruber/McKnight study finds that plans limiting patient choice to particular (cheaper) providers can hold down total spending. No surprise there. What about the title of the Vox article though? What does the Gruber/McKnight study say about quality of care? Here’s where things get interesting.
The Vox article correctly summarizes the paper. In the interest of brevity, I’ll reproduce just the key quotations from the Vox piece:
Gruber and McKnight’s research takes advantage of a natural experiment that happened in Massachusetts three years ago. Back then, the state gave some employees a big financial incentive to switch to narrow network plans: three months of free premiums. The researchers could look at the two groups from 2010, before the switch, up through a year after the change in 2012. Here’s what they found.
1) With the three-month premium holiday, lots of people switched to the narrow network plan…
2) Health spending fell by one-third for those who switched….
3) Patients used more primary care — and got fewer specialty services. Patients who switched to the narrow network plans spent 28 percent more on trips to primary care doctors — but 45 percent less on specialty care. They went to the emergency room less and got fewer x-rays and scans.
…While urgent care clinics and emergency room travel didn’t change, ***trips to the hospital became much longer, by as much as 40 miles.***
4) The narrow network hospitals provided just as good care. This is always a big concern with narrow networks: the health insurer picks the cheapest hospitals, which are in turn the worst hospitals, and patients end up getting worse care.
At least in the Massachusetts example, this wasn’t the case. Gruber and McKnight look at a typical panel of health indicators, like how many heart attack patients die within a month and re-admission rates where the hospital messed up the first time.
They find that hospitals in and outside of the narrow plans look pretty similar; there’s no significant difference in quality. “Enrollment in limited network plans is not associated with any change in the quality of accessible inpatient hospital care,” Gruber and McKnight conclude.
***Gruber and McKnight were unable in this particular study to look at the health outcomes of patients — to explore whether limited network enrollees fared better, worse, or the same in the care of a smaller handful of providers.***
I have put asterisks in the above quotation to highlight the critical pieces. Because they said they lacked the necessary data to make precise comparisons, Gruber/McKnight didn’t actually look at the health outcomes of the people who participated in the new network plan (which restricted patient choice). They admitted that some people saw an increase of 40 miles to the hospital they would need to use.
Now when Gruber and McKnight say there was no change in “quality of accessible inpatient hospital care,” they are looking at the people *who actually are treated at the hospitals in question*.
In case it’s not clear why this is an entirely different issue, consider the following analogy:
Suppose Goldman Sachs thought its executives were abusing their travel budgets. In order to weed out the frivolous flights, Goldman thus institutes a new policy: Any executive who agrees to only fly out in and out of Reagan airport—located in DC—will get a $5,000 bonus. Executives who wish to retain the option of being reimbursed for their work flights using JFK or LaGuardia may continue to do so, but they forfeit the $5,000 bonus.
Now what would happen if we checked on the groups who did and did not take the bonus money a year after the new policy had been implemented at Goldman Sachs? We would naturally find that the executives who took the money ended up requesting much less in travel reimbursements. After all, if they wanted to fly somewhere for work, they would have to book the flights through DC, meaning they would need to take Amtrak or some other mode of transport down to that airport, rather than the much more convenient NYC-based airports. And indeed, researchers would also verify, after the fact, that the executives opting for the restricted-choice air plan ended up making fewer trips to the airport.
However, what would not be so obvious is that this change would—apparently—have no adverse effect on the quality of travel. The researchers would look at various metrics, such as flight delays, crash statistics, lost luggage, and so forth, and not see any appreciable differences among the two groups regarding DC or NYC flights. There were certainly no differences so strong that they would overwhelm the $5,000 bonus payment. (The researchers admitted in a footnote that they didn’t actually look at the total travel time, satisfaction with work trips, productivity per travel dollar spent, or other metrics for the executives who opted for the plan; such data was unavailable at the time of the study.) Thus after this initial experiment, Goldman Sachs would make participation mandatory, forcing all of its executives to eschew NYC airports and always depart from DC, since it had empirically verified that this policy was all gain, no pain.
Does everyone see the huge methodological problem with the above study? Can you imagine if Vox ran an article summarizing the new research, proclaiming, “Limited airport choice plans don’t mean worse travel“? If the federal government then foisted such a plan on all major employers, the people who had read the Vox piece would be surprised to discover that they suddenly found it a real hassle to fly. Suddenly they would see exactly what the study authors meant by the “quality of accessible air travel.”
Facebook
YouTube
RSS