Musings of a Cancer Doctor

Wide-ranging views and perspective from George W. Sledge, Jr., MD

Monday, April 15, 2013

On Scientific Journals

Last week I found myself at one of those small side meetings that attach themselves, remora-like, to large national meetings. The large meeting was the American Research Association for Cancer, but it could just have easily been ASCO or San Antonio, or any of those other road shows where scientists congregate. I heard a wonderful presentation from a prominent lab scientist, who then went on to bemoan the fate of the work in the hands of the (arguably) world’s leading scientific journal.

The reviewers had said that the work was revolutionary, new paradigm stuff, and a real game-changer. Because it was such extraordinary and unexpected work, they wanted every experiment in the paper independently repeated by another lab. And the scientist had tried, getting about 60% of the work independently confirmed. The editors had still refused to publish it, and it ended up in another (quite excellent) journal, two years after initial submission.


This one experience tells you so much about the modern scientific journal: the academic’s need and desire to get his or her work published in the most prestigious journal, or at least the most prestigious journal you think will take it; the journals’ desire to be selective and exclusive, pursuing the all-important high citation index; the often arbitrary nature of the review and the sometimes extraordinary hoops one is forced to jump through; the long delays and sometimes endless cycles of revision; the anger/despair associated with the rejection of the work; and the odd way in which every work eventually finds its own level.


What was even more curious was that, even as the scientist was presenting this work to our small group, I was aware of a colleague who had independently discovered essentially the same results, with the same outcome of rejection. How often do such things happen? I personally cannot claim to have discovered something earth-shattering that never saw the light of day due to editorial intransigence, but over the years I have certainly heard the same lament from many colleagues: how their work ran into some editorial buzz saw that left their bleeding corpse of a research project to die unnoticed, while others published similar work to general acclaim.


I sympathized, but I’ve also been on the other side of the fence, both as a reviewer and an editor. The sheer amount of crud that floods editorial offices has to be seen to be believed, and even the good stuff is usually improved through editorial review. I believe in peer review even though I recognize its limitations.


Date Back to 17th Century

Scientific journals date to the 17th century. The very first, the Philosophical Transactions of the Royal Society, has been in continuous existence since 1667. It was, arguably, the dividing line between the ancient and the modern worlds. Before its creation, knowledge was supposed to be arcane: something mysterious, hidden, and known only to a few cognoscenti.


The scientific journal changed all that. The first great scientific discovery to be published in a periodical was Isaac Newton’s 1672 experiments on white light, in which he used a prism to demonstrate the spectrum. Experiment as metaphor: the blinding white light of science reveals a multi-colored collection of knowledge, for all to see and know. Suddenly, and quite unexpectedly in historical terms, knowledge was democratized. It was a revolutionary moment, and nothing has ever been quite the same since. The modern world, a world created by human science and technology, can be drawn in an almost straight line forward from the invention of the scientific journal.


Well, you can have too much of a good thing. Even chocolate can give you a tummy ache if you eat too much, and we have certainly swallowed too much of scientific journals. Let me count the ways: there are too many of them, they cost too much, it is impossible to keep up with all of them, they reflect the biases as much as the wisdom of the editors and reviewers; the review process is secretive and non-transparent where science is supposed to be open and transparent; they are ineffective at preventing scientific fraud; they publish much that is irreproducible, and they fill up my office with unopened piles of unread paper that add to global warming by cutting down great forests. I think I can document every statement in this paragraph, and indeed others have written learned papers on the subject (thereby chopping down more unoffending trees), but for the moment just take my word for it.


Financially, scientific journals represent an interesting economic case. Scientists work on a project paid for by taxpayers or some foundation. They then submit it to a journal, where an editor farms it out to unpaid reviewers (the same ones over and over again: you learn to rely on compulsive masochists in the editorial biz). A percentage of submissions are then published in journals that are priced like some tiara at Tiffany’s. The copyright then belongs, not to the authors, but to the publishers, in perpetuity. Even decades-old articles are held hostage behind paywalls by publishers looking to wring every last cent out of work paid for by others and vetted by unpaid scientists.


How pricy are scientific journals? Nature recently reported (2013;495:426-429) that the scientific publishing industry generated $9.4 billion in revenue in 2011 and published 1.8 million English-language articles, or about $5,000 per article. Profit margins for the industry are estimated to be in the 20-30% range, with margins varying by publisher type: 20% for society publications at the low end, 35% for commercial publishers at the high end. Publishing articles about new drugs has a higher return on investment than making new drugs, and requires considerably less creativity. Think about it.


And like new drugs, the economics are ultimately unsustainable -- albeit the cost of bankrupting academic libraries pales in comparison to that of bankrupting Medicare. Academic libraries frequently spend 90% of their budget on journal subscriptions, even as journals continue to proliferate like randy rabbits. Between 2007 and 2011, while the rest of the world was mired in the worst recession since the Great Depression, the number of medicine and health-related journals grew by 19%, according to a report by the Office of Science and Technology Policy.


Scientific publishing is evolving rapidly, as the transition from print (carried by snail mail) to digital formats continues. Will scientific articles even be printed five or 10 years from now? I suspect that the main reason many journals remain in print format is for the attached advertisements, which are far more lucrative than online ads. It isn’t just commercial publishers who benefit from these advertising revenues: medical societies rely on them to eke out an existence. I know that when I was ASCO President we religiously followed quarterly ad revenues, and I suspect those who followed me do the same.


Recent changes have revolutionized scientific publishing. To name but a few: open-access journals such as PLoS ONE are taking off, and now account for some 11% of publications. Some institutions (Harvard, for instance) require faculty to deposit articles in free repositories open to all. PC’s store libraries of PDFs obtained through Internet searches, ablating countless visits to the library. In some fields (high energy physics is perhaps the best example) scientific findings are widely distributed online (though sites like arXive and ResearchGate) before submission for publication. Most journals have, in recent years, added “epub ahead of print” to get the word out months before the plastic-entombed pre-mulch arrives by snail mail. Online-only supplemental data has hugely altered the volume of data available to interested readers, a real plus (or minus: take your pick) in the genomic era.


Research blogging, increasingly common, serves as a simplifying conduit for (and sometimes critical evaluator of) our work. Taken together, these changes amount to a transformation in the transmission of scientific output from researchers to the scientific public.


New mechanisms of peer review are being experimented with as well. Open access  (OA) has found its parallel in open evaluation (OE), where peer review continues post-publication, all reviews are open to the public (as opposed to the current process of secret ratings available only to the editor), and the reviews themselves are meta-evaluated. Sound strange or unlikely? Well, if you have ever seen the online Rotten Tomatoes movie reviews, which give a numerical score for each movie derived from such a meta-review, you get the idea -- Not possible a decade ago, but trivial in a Web-based era, and a very democratic way to view scientific output.


We seem to be at some sort of inflection point in scientific knowledge, the event horizon between a slowpoke past and a hyper-dense, rapidly moving data future. Asking what role scientific journals will play in that future is another way of asking how we will organize our knowledge, and how will we transmit it, and perhaps even who we will transmit it to.


I read the other day about computer programs that score college entrance essays. I don’t doubt that 10 years from now, once text mining and other analytic algorithms are in place, the computers will be judging our scientific submissions as well. Maybe even pre-reviewing the data as it comes out of other machines in the lab. Peer review? Forget that: our “peers” will share opinions about us in silico. We’ll have something new to complain about when we get the reviews back. I hope they still let us go to our meetings.