Last week the journalism world was buzzing about — guess who? — Jonah Lehrer. Yes, again. We knew about the science writer’s self-plagiarism and Bob-Dylan-quote fabrication. Last week a New York Magazine exposé by Boris Kachka claimed that Lehrer also deliberately misrepresented other people’s ideas.
Kachka’s piece led to some fascinating discussions about whether it’s possible to tell a science story that’s both riveting and fully accurate. Science journalist Carl Zimmer, for example, wrote a thoughtful, inspiring post about the messiness of science. All of the commentary left me wanting to hear more details from the scientists in Lehrer’s stories. Had they been misrepresented? If so, how? Were they upset? Did they complain?
Kachka and Zimmer zeroed in on a 2010 story about the scientific method that Lehrer wrote for the New Yorker. The story’s premise is clear from the title (“The Truth Wears Off”), the subtitle (“Is there something wrong with the scientific method?”), the nutgraf (“It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”), and the last few lines (“Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.”).
This article works well in Kachka’s piece for two reasons. First, it has a dash of got-cha! Schadenfreude: Well well well, fabricator Jonah Lehrer doesn’t believe in truth! The second reason, and the one I care about, is that the article was vetted by the New Yorker‘s famous fact-checkers. It has no tweaked quotes, no plagiarism, no obviously wrong statistics or data points — in other words, none of the easy mistakes that all of us make, occasionally, and might be forgiven, occasionally. Lehrer’s offense in this piece, at least according to Kachka, is worse than any of that. Lehrer describes an idea — The scientific method is broken — as if many scientists support it. Do they?
The scientists quoted in the piece are interested in the ‘decline effect’, the phenomenon that robust scientific results often can’t be replicated. It’s a fascinating topic, with enormous implications, and worthy of attention by the New Yorker. But is the decline effect, as the article claims, a rebuke of the scientific method?
On Friday I wrote to seven of the scientists quoted in the article and asked what they thought of it. Six responded, with the earnestness and precision that I’ve come to expect from scientists. The short story is: Almost everybody was satisfied with Lehrer’s specific descriptions of their work, but some were annoyed by his spin on it.
Some excerpts (with spelling and grammar cleaned up):
Rich Palmer: I had no problem with Lehrer’s portrayal of my own results…I actually thought he did an excellent job there. I could take issue with his subtitle “Is there something wrong with the scientific method?” (which so offended Kachka), because the problem isn’t with the scientific method in the strict philosophical sense of “observation –> hypothesis –> test” or “one chooses between alternative hypotheses based on new data”. The problem arises because the ‘scientific method’ is implemented by humans, and humans are inherently fallible.
Part of my email had read: “Was the piece, in the end, true?” Palmer wrote:
There is no truth (except in mathematics). All other so-called scientific conclusions are necessarily false to some degree (i.e., include some uncertainty). I think journalists and bloggists need to make peace with this.
…If there’s a lesson here, it’s about a widespread human failing. Most people would rather some other clever person distill down all the complex details into a good story for them, preferably in excellent prose. But those distilled stories should never be treated as a substitute for original research results. If anyone really wants ‘the truth’, they’re going to have to slog through an awful lot of turgid and arcane original research and draw their own conclusion.
Leigh Simmons: I actually thought that piece was rather good. And he quoted me accurately.
John Ioannidis: I think that Lehrer’s write-up was overall pretty good. Retrospectively, perhaps the one aspect that seemed a bit off was the tone, but this is not an unusual problem in journalistic writings (unfortunately it is very common even in purely scientific writings).
In a follow-up message, I asked Ioannidis if he agreed with the somewhat mystical argument that the scientific method doesn’t work. He replied:
Of course, I certainly think the scientific method does work (!). It is primarily an issue of optimizing its efficiency and making sure it is applied the way it should be applied. Retrospectively, perhaps you are right that there was a bit of mysticism in the ending of that piece, but I thought it was primarily a figurative mode of writing that you would expect from New Yorker rather than from a Science or Cell or JAMA article.
John Crabbe: Well, I disagree with the conclusion that science is broken. But it was, after all, an opinion piece. I felt that Lehrer’s portrayal of our study was partially accurate, but also misleading.
…Lay science is a very tough game, I think. You have to simplify, and you have to engage. This leads sometimes to provocative lines of argument. While I am not at all opposed to this, I do think he went quite a ways beyond the data in his NY piece. I wouldn’t characterize it as scientific misconduct…He didn’t invent anything, just spun complex data.
Jonathan Schooler: Lehrer appropriately portrayed both my research and the discussions that I had with him.
…Lehrer’s conclusion that “When the experiments are done, we still have to choose what to believe” suggests that in the end science is simply reduced to opinion. Although I do not personally concur with this view, I do not think that in stating this perspective Lehrer was misrepresenting the scientific views that he queried. Lehrer ended with his own editorializing on the subject and while I do not personally endorse his final opinion, he never suggests that I nor other scientists necessarily subscribe to his view.
Michael Jennions: I think his portrayal of my work was perfectly fair…One of the charges levelled against Lehrer with respect to his article is that he implied that science might be broken. The last paragraph in particular annoys many scientists…Perhaps Lehrer should have emphasised that these are not mainstream views?
…This leads to the more general problem. If scientist A says X occurs, and scientist B says X does not occur – what should the journalist do? Is their job to be balanced and present both cases, or to work out where the balance of evidence lies and reach a conclusion? This is an age-old question that all journalists struggle with, and there are rarely simple answers.
That age-old question is a good one. Would it have been preferable for Lehrer to cover the controversy — that is, to not only explain the decline effect and its various explanations, but to point out to his readers that some of the explanations are more widely accepted than others? This would have made for a more complicated narrative and, perhaps, a less popular one. If he had pitched the story that way, would his editors have bought it?
I also asked the scientists whether they were upset about the article, and if so, whether they did anything about it. Some, like Schooler, were happy: “I was not upset, to the contrary this article accelerated discussion of these important issues.” Ioannidis simply shrugged: “This issue certainly did not reach the level of annoying me to the point that I would think a rebuttal would be due.” Crabbe, though, was pretty miffed. He wrote Lehrer an email, received a reply he wasn’t satisfied with, and ultimately wrote a formal complaint letter to the magazine. Here’s part of the letter that resonates strongly with me:
“To me, a working scientist, your article was troubling not because of the solid points scattered throughout (Science is difficult, measurement itself is difficult, publication bias is a problem, etc.). Rather, the message to a non-scientist (the large majority of New Yorker readers) is, science is a useless exercise. Nothing can ever be learned from it because no single experiment can establish ‘truth.'”
I happen to agree with Crabbe’s assessment of the New Yorker story. But the thesis isn’t the story’s biggest problem. The biggest problem is that Lehrer presented an argument that is not supported by the vast majority of scientists, and never let his readers know just how far out of the mainstream he was taking them. I think it was misleading, and perhaps dangerously so.
Hearing from these scientists made me wonder, somewhat despondently, whether science journalism is a useless exercise. Here is Lehrer, one of the best science writers I ever read, publishing in the most elite magazine with the help of the smartest editors and most rigorous fact-checkers. And still, still, the story isn’t true. Why should the rest of us bother, and why should scientists give us their time in our attempts?
In another email, Crabbe gave me a damn good answer. “Science journalism is crucial for the maintenance of a funding base for science,” he wrote. “Why should people pay for stuff they don’t understand at all!” There’s also the broader aim of helping people better understand the natural world. Lofty, maybe, but worthwhile goals.
And yeah, it’s hard, but then so is science. I suppose, like these scientists trying to learn from their failed experiments, we science journalists just have to keep trying to root out the truth.
I invite Lehrer and all of the scientists mentioned here to leave a comment if they’d like to say more — I only quoted parts of their responses, after all. I especially hope they pipe up if I’ve misrepresented their views, which would be so deliciously meta, wouldn’t it?
A big thanks to fellow science writer Kelly Rae Chi, whose Twitter messages inspired this post.