In Jonah Lehrer’s recent New Yorker piece, The Truth Wears Off, he discusses Jonathan Schooler’s idea that science is troubled by a disturbing trend of publication bias that has resulted in statistical noise being reported as significant new findings. When repetition accounts for that noise, the “Decline Effect” as Schooler calls it, makes those initial significant findings disappear into the ether.
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe?
“For many scientists, the effect is especially troubling.” Is it? This regression toward the mean is exactly why scientists perform follow up studies in the first place. This is the raison d’etre for the scientific process as we know it today. If repetition shows an earlier conclusion is in error, then it was not so “rigorously validated” now was it? The truth will out. Time makes fools of us all. (E.T. Bell)
No scientist should be disturbed by the trend. If anything, it should reaffirm their faith in the process, and Lehrer never returns to admit that, even in the piece’s conclusion.
The decline effect is troubling because it reminds us how difficult it is to prove anything.
This is not news to anyone. (Also, be careful of how you use that word prove when discussing science. This isn’t a court room.) In fact, that difficulty is why people trust the results that have been rigorously tested. The Decline Effect does not indict the scientific process, Mr. Lehrer. It indicts the scientific publication process, and scientific journalism that reports the hype. (Just look at the recent “life on other planets” press conference nonsense.)
There is a bigger problem that cuts to the heart of this issue that neither Schooler nor Lehrer addresses here: Short term gain vs long term viability. Science with a capital S, proper noun, is obsessively fixated on the long term. Time, repetition, confirmation or refutation: these are the things that make it what it is. The process removes human error through repetition and regression toward the truth.
The fact that a great deal of research is done by corporations, in particular pharmaceutical companies, is no surprise. Companies need to prove their products and processes, and the scientific process is the best way to do it. The corporate-backed science regime is a double edged sword, though. The profit motive and long term economics actually reinforces the objectives of good science. Measured over time, accurate information is simply more valuable than misleading information. Companies behaving responsibly should want good information.
The problem arises when short term gain is elevated above long term viability. The U.S.S.R.’s crises were worsened because they were systematically reporting bad information. It wasn’t communism that failed, it was institutionalized lying.
The political structures governing medical treatment in the U.S. are similarly rewarding for companies who think in the short term. The sooner they get approval, the longer they can exploit their patent before generic sales begin. In the long term, generic sales will be lackluster if the drug is ineffective, but by the time those studies confirm that the drug is ineffective, the company has already gotten a decade’s worth of hyped-up brand name sales and is effectively “done” with that drug.
The same trend exists in energy (dirty vs clean energy), stock, bond, and currency markets (sub-prime lending and credit default swaps are mathematically impossible to make viable in the long term), and many other things.
When immediate survival is threatened, short term survival is prioritized appropriately. But when long term success is clearly achievable, but short term gain is rewarded, a depressing number of people still opt for short term gain.
The trouble with corporate science in the U.S. actually extends beyond the lab, and even beyond the company. The problem is that the current tax structure makes a stock’s closing price more important than the actual underlying asset: the company’s profitability and long-term viability. As a result, business decisions affect how science is reported, and the selection bias toward statistical noise.
This is short term vs long term problem is not unique to science. But science at least has a mechanism for dealing with it. What we, as a society, do with unconfirmed scientific results is not to be blamed on the scientists themselves.