Here’s a little snapshot on the state of junk science and junk science reporting. Most articles are from September 2012, as this topic has received a lot of attention lately.
Alok Jha’s False positives: fraud and misconduct are threatening scientific research offers a grand review of these lamentable issues, especially in psychology and other medical disciplines. Discovery is often accidental, and only after the fraudulent papers have been quoted and relied upon for years. Replications of previous studies are unattractive from a career perspective, and so are negative results. Failed replications are unlikely to get published, as journals prefer sensational new positive results – true or not.
Chambers & Sumner reply that replication is the only solution to scientific fraud. That requires both the raw data and the experimental methodology being publicly available, and ideally an incentive structure rewarding scientists whose work could be widely verified. The present situation is far from this ideal: Joshua Carp found that most brain imaging studies lack the methodological detail required for replication. Craig Bennett and colleagues showed brain activity in dead salmon – a nice demonstration of what nonsense can result from cherry-picking the most favorable statistical outliers.
What about the vaunted peer review system, supposed guardian of academic quality? Daniel Lemire notes that most rejected papers are accepted elsewhere, and at any rate reviewers find it hard to reach consensus on a paper’s quality. Back in April, Carl Zimmer reported on a sharp rise in retractions even at well-regarded journals. And Uri Simonsohn, operating on the principle that results too good to be true probably are, discovered several academic frauds by their improbably accurate or consistent data. That included the famous case of Yoshitaka Fuji who had fabricated results in 172 (!) studies published in peer-reviewed journals over 18 years – despite fraud allegations as early as 2000!
Compounding the faults of junk science are those of bad science journalism. Seth Mnookin recounts the state of science writing, circa 2012, including the self-destruction of Jonah Lehrer but also various pieces in prestigious newspapers that were guilty of sensationalist exaggerations or outright nonsense. Citing a study on ADHD reporting he concludes: We pay lots of attention to things that are almost assuredly not true.
Only the most sensational initial claims are reported; future refutations or attenuations are ignored.
Journalists are not entirely to blame, however. A small-scale study found that sensationalist spin usually originates in abstracts, i.e. with the scientists themselves. And then we have scientists requiring an embargo agreement from journalists, forbidding them to consult other experts before publication! Unsurprisingly, the paper (alleging cancer in rats from genetically modified food) turned out to be extremely dubious. Carl Zimmer recalls a similar 2009 incident involving an overhyped primate fossil, and Athena Andreadis notes that the shoddy ENCODE reporting was partially caused by deliberately staging a “media blitz” after years of embargo. Once more Seth Mnookin:
So, to summarize: one of our biggest stars was revealed as a fraud; publications that should be exemplars of nuanced, high-quality reporting are allowing confused speculation to clutter their pages; researchers and PIOs are nudging reporters towards overblown interpretations; and everything we write about will probably end up being wrong anyway — not that we’ll bother to let you know when the time comes.
But Mnookin does point out the silver lining that’s also evident from this post: we can get proper information on science and scientific publications today, thanks to honest experts writing on the Internet. Even as the gatekeepers are failing us, we need them less than ever.