ARTICLE IN BRIEF
Neurology editors, past and present, debate the merits of the open access publishing model in light of an investigation that found a nearly 50 percent acceptance rate among open access publishers of a sham research article.
The saga reads more like an undercover detective caper than scientific research: a crusading writer for Science magazine, investigating the quality of the review process at open-access scientific journals, concocts a phony research paper claiming to uncover the anti-cancer properties of a chemical extracted from lichen.
The “lead author,” as well as his research institution, does not exist. But the paper is carefully crafted to be plausible while at the same time contain obvious flaws that any good peer-review process should detect, including a graph with a caption claiming that it shows a “dose-dependent” effect on cell growth, when in fact the graph reveals no such thing. To make the non-English-speaking “author's” writing more believable, the paper is run through Google Translate forward and backward.
The result: Nearly half of the 304 open-access journals that received the phony “research paper” accepted it. Another 94 rejected the paper, and the remainder hadn't taken action by the time the article went to press in Science in early October.
Some of these journals appeared to be fairly fly-by-night operations, but other publications that accepted the phony paper included those run by respected publishing houses such as Elsevier, Sage, and Wolters Kluwer, which produces Neurology Today.
The Oct. 4 Science article — some have called it an “investigation,” while others consider the term an overreach — written by well-known biologist and science journalist John Bohannon, has set off a firestorm of debate in the general media as well as, more colorfully, in scientific blogs like Genes Gone Wild (http://bit.ly/HNi7F4) and Michael Eisen's it is NOT junk (http://bit.ly/M79cc). In essence, the question for debate is this: Has Bohannon revealed an inherent flaw in the open-access model itself, or does the problem lie not with the publishing model, but with the quality of peer review at any scholarly publication, open-access or not?
MIXED RESPONSE FROM NEUROLOGY
Neurologists are no more unanimous on this question than the scientific community in general. Some are highly dubious about open-access, and see the Bohannon paper as further evidence of its unreliability.
“I think we are now reaping the crop of the sowing of the seeds of replacing peer-reviewed journal with an open-access model,” said Robert C. Griggs, MD, professor and former chair of the department of neurology at the University of Rochester Medical Center, who edited Neurology for ten years. “Authors are expected to pay for reviewers and reviewers are expected to review, mostly without pay. Journals like Neurology are rigorously run and reviewed; while they make mistakes occasionally, as editor, I looked at every review and every paper myself.”
Dr. Griggs said that he gets about one invitation a week to serve on an editorial board, or even act as editor, for an open-access journal. “I've never accepted one, because of the problems they represent. Any time you replace a system that's worked with a system that hasn't been adequately tested, you shouldn't be surprised if there are problems.”
Dr. Griggs acknowledges that some open-access journals have a better reputation. “PLoS One does a good job by and large,” he said. “And Annals of Neurology has started an open-access journal. While I can't say for certain yet, I assume that it will be rigorously reviewed and be a reasonable place to put papers.”
Nonetheless, he believes that the economic model underlying many open-access journals creates an unavoidable conflict. “If you pay your money, you get your papers published,” he said. “There's a pressure on everyone to accept a paper if the author's going to pay once it's accepted.”
WAS THE SCIENCE ARTICLE FLAWED?
But many have criticized Bohannon's article as, ironically, not sufficiently rigorous in its investigation. If he is studying the potential flaws in peer review in the open-access publishing model, they argue, shouldn't there be a “control group” — that is, shouldn't he have submitted the same paper to traditionally-published journals to see how they fared in spotting the fake?
“This paper itself was methodologically flawed,” said John “Jack” Kessler, MD, Ken and Ruth Davee professor of stem cell biology at Northwestern University and the editor-in-chief of Annals of Clinical and Translational Neurology, a new online-only, open-access journal published by Wiley for the American Neurological Association. “There was no control group. If the goal was to determine whether the publishing model was responsible for the shoddy review process, then the paper also should have been sent to journals with the traditional publishing model. Science published a flawed paper. It should not have passed a review process that asked the basic question of whether the conclusions were supported by the data.”
At press time, Bohannon offered a detailed rebuttal to these and other criticisms of the methodology behind the Science article in an interview posted online for the Scholarly Kitchen, a blog of the Society for Scholarly Publishing. See “Back and Forth on Open Access” for Bohannon's response.
That said, Dr. Griggs has strong words for the journals that allowed the Bohannon paper to pass their review process. “Everything that's written about that particular article indicates it shouldn't pass even a cursory review, much less a careful one,” he said.
“The article makes clear that there appear to be so-called ‘predatory’ OA [open access] journals that seem to be in the business for profit only,” agrees Robert A. Gross, MD, PhD, professor of neurology at the University of Rochester Medical Center and current editor-in-chief of Neurology.
But Drs. Kessler and Gross believe that these screening failures are an issue of peer review, not of the publishing model. “Open-access is a newer phenomenon than the standard subscription model,” Dr. Kessler said. “Therefore, there are a greater number of newer journals that are less well established. And not surprisingly, many of these are likely to have papers with lower impact than those in journals that have been around for a longer period of time. But the review process of a quality open-access journal will be just as rigorous as the process in traditional journals.”
THE ‘PAY TO PLAY’ MODEL
What about the “pay to play” model that characterizes many open-access journals? Isn't that an undue influence on editors to publish shoddy work in order to reap the author fees?
“It is true that many OA journals are based in publishers that appear to be in it solely for the money, something the article made very clear,” Dr. Gross acknowledged. “Their processes regarding peer review were weak, to be sure. It is also possible that the ‘pay-for-publication’ model incentivizes publication over rigor in the review process, though this would depend largely on the publishers' and editors' standards.”
But subscription journals may have their own undue influences. “Every journal has to earn its keep,” Dr. Kessler pointed out. “Subscription journals have to sell enough copies. Journals do many things to increase their impact factor, because that metric seems to have gained a lot of currency. Many publish ‘review’ articles because those seem to be cited more frequently than standard research articles. Many journals with high impact publish things that will attract media attention even if they aren't of the quality that would be expected. Every scientist knows that there are outstanding articles in journals with less of a reputation and weak papers that cannot be replicated in journals with a strong reputation. Is that a castigation of the subscription model?”
Some journals can afford to be more rigorous than others, Dr. Gross observed. “Neurology currently rejects the majority of submissions, so we can be very rigorous in our review process. Journals that need submissions to fill up their pages may be incentivized to review less rigorously.”
What about open-access neurology journals in particular? There are currently 146 open-access journals related to neurology, according to the online Directory of Open-Access Journals (DOAJ). Annals of Clinical and Translational Neurology (ACTN), which didn't yet appear in a Nov. 5 DOAJ search, would make the total 147.
Dr. Kessler said that ACTN aspires to uphold the same standards set by the ANA for decades. “We are not going to publish anything less than solid papers,” he declared. The journal's first two papers have just been published, and so far its rejection rate is about 65 percent. “That's because we have a responsibility for very careful review.”
The bright lines in scientific publishing between “open-access” and “subscription only” may be fading anyway. “Neurology, for example, is a hybrid journal, offering OA as an option,” said Dr. Gross. “We did this in response to many funders requiring this of their investigators. Why require it? Well, OA offers the widest possible distribution of the material; there are no barriers. Our peer-review process is the same for OA or subscription access papers. The editors and reviewers handle these papers the same way.”
The National Institutes of Health now requires that anything created with its support be made open-access within six months of publication. “That's changed the ballgame right there,” said Dr. Kessler.
Other new online models may offer what some have called “post-publication peer review.” One of these is “WriteClick,” Neurology's rapid online correspondence posts section, overseen by Dr. Griggs. “If people find that a paper has a mistake or overlooks something, you can publish something to correct it within two or three days,” he said. “We have had a number of people write in having found mistakes, and that's corrected papers very rapidly. I don't think it's led to a retraction as yet, but we came close to it with one author who had to say that his data were reliable, but his conclusions were invalid.”
The fact that post-publication peer review is necessary, even in highly respected journals, highlights the fact that peer review is not perfect. “It's a human process, and humans have biases that can, at times, be subtle or hidden, especially to those having them,” said Dr. Gross. The job of reviewers is to make sure the science and its interpretation is sound. The editors should assure the overall rigor of the process. Mistakes and oversights will happen, however; it's inevitable.”
But that's not a function of the publishing model, he said. “Open access is a business and distribution model, not an editorial model. Its incentives may work toward less rigor, but high editorial and publisher standards should be able to address this.”