That we make progress in cancer science is certain, but progress is slow and sometimes uncertain. Some of our failure to understand the biology of malignancy, and apply this biology to better therapy, is related to the undeniable fact that cancer is hard. But some of the setbacks are self-inflicted wounds. If medical science were a person rather than an enterprise, that individual would be bleeding from multiple sites, poisoned, and burned. That it keeps going points to the importance of the subject and the forgiveness of the general public.
Let’s do a little CSI work. The first, and flashiest, self-inflicted wound is fraud. Every few years we have a major fraud in cancer research. During my career, names such as (in reverse chronologic order) Anil Potti, Werner Bezwoda, and Roger Poisson come to mind. Potti damaged the reputation of a great university (admittedly with that institution’s willing compliance) and harmed the emerging field of genomics. Bezwoda poisoned the well for high-dose chemotherapy and autologous bone marrow transplantation, and in the process hurt many real patients. Poisson came close to destroying one of the world’s great cooperative oncology groups.
I met two of the three (Bezwoda and Poisson), and just missed the opportunity to meet the third (Potti) at our local oncology grand rounds (where he was introduced, so I am told, as a Rhodes scholar, one of his lesser malfeasances). Bezwoda, who I met before his fraud was exposed, struck me as defensive, moody, and unpleasant, though I did not suspect him of fraud. Poisson, who I met after he torched the NSABP, struck me as a well-meaning man in over his head.
Science is self-correcting, as are all democratic institutions, so we can survive the occasional fraud. External clinical trials audits exposed Bezwoda and Poisson, and sharp reviewers and a fine reporter figured out Potti. The occasional big fraud is usually exposed, sooner or later, because it attracts attention. And while it endangers the lives of patients (truly unforgivable) it rarely does violence to the process of science.
In the short run fraud can cause the river of research to change direction briefly, but usually only slightly. Genomics progresses on its merry way despite Potti: there are just too many investigators, too many cool new toys, and so much neat data. Bezwoda’s positive adjuvant breast cancer trial was presented along with several other negative randomized trials, the discordance prompting the career-ending audit of his program. The NSABP, being a large, multi-center organization, survived the loss of data from Poisson’s institution in the affected trials, whose overall results remained unchanged, though the organization’s recovery was slow.
My major concern with fraud is how long it takes us to catch on. The story with fraud, in both lab and clinic, is that no one ever does it just once. Most of those committing fraud are serial liars, and not even particularly convincing liars at that, at least not in retrospect. Why are we so slow to catch on? Why does the peer review process miss them, time and time again, until they consider themselves untouchable and commit a true howler?
A second self-inflicted wound involves poor ingredients. Great science, like a great meal, depends on the ingredients, and there are times when the ingredients that go into the broth just stink. In cancer science we rely to an incredible extent on cell lines for in vitro and xenograft experiments. But these cell lines are either of poor provenance, or contaminated, to an amazing degree. There is an interesting recent article in the Wall Street Journal by Amy Marcus on this subject, which recounts the familiar subject of HeLa cells (which can grow in the vacuum of deep space, apparently, and contaminate every lab they come in contact with).
The other story in the article is less well known outside of breast cancer circles. MDA-MB-435 was, until a few years ago, the poster child cell line for triple-negative breast cancer. It grew quickly in vitro and in nude mice, routinely and reproducibly metastasized to the lung, and as such served as CV polymerase for many breast cancer investigators.
I know, because I was one of them. I published papers in Cancer Research and the Journal of the National Cancer Institute based on MDA-MB-435 work, and one of them was one of my better-cited papers. So I was personally quite disappointed when it turned out that it was, to all intents and purposes, a melanoma cell line.
As the Wall Street Journal article makes clear, there are still arguments as to whether the cell line is a breast cancer weirdoma (honest but irrelevant to the greater breast cancer problem), a contaminant (a cell line con operating under false pretences), or cells from a patient who had both breast cancer and melanoma. I have no particular bias, but as soon as I was certain the cells acted like, looked like, smelled like melanoma cells, I stopped working with them. Like a chef who discovers that his $220/ounce black truffles are common mushrooms, stinky rather than aromatic, I tossed them overboard and never looked back.
A self-inflicted wound (well, I didn’t inflict it, but the scientific system I belong to did) cost me a bunch of time, and the time of many others. If those who read my paper then initiated work based on it -- a reverberating effect of the bad ingredient -- I regret the time they wasted as well. I suspect there are many other bad ingredients out there: other bad cell lines, insufficiently pure mouse strains, and (especially common) lousy antibodies against rare and not so rare antigens.
We are all at the mercy of bad ingredients. When, in my lab, I find that an analytic antibody doesn’t work, I stop working with it and find another that does. Time and money wasted aside, there is no “bad ingredient central clearinghouse” where one can report analyte failures. Maybe we need one, an Angie’s List for antibodies and cell lines.
The final self-inflicted wound, however, is the most insidious and dangerous one of all, because it is the most common: the one caused by taking shortcuts in research.
Glenn Begley and Lee Ellis recently wrote a commentary for Nature (2012;483:531-533) that is well worth your time reading, and somewhat frightening in its implications. Investigators at Amgen, looking at 53 “landmark” studies in the preclinical literature, were able to confirm the main findings in only 11% of cases. A similar study from investigators at Bayer demonstrated that only 25% of studies could be validated.
There are many reasons one might be unable to confirm another’s results. I’ve learned that “Materials and Methods” sections of papers often leave out some critical step or ingredient. I’ve already mentioned the “bad ingredient” problem, which would obviously affect validation.
But Begley and Ellis point to the basic sloppiness of much of the work being published today: cherry-picking of data, lack of reproduction of critical experiments, and use of single cell lines to represent the breadth of a particular disease (imagine using only a single patient to represent all of breast cancer or colorectal cancer when testing a new hypothesis). And, underlying everything else, a system that encourages getting published first rather than getting the science right.
In short, a system built on shortcuts. This system is now responsible for (as recent data suggests) an all-time high in retractions of papers. Indeed, the higher in prestige a journal, the more likely one is to see a retraction of a peer-reviewed study.
This doesn’t imply or require willful fraud; never assume maleficence when simple laziness or incompetence will serve as explanation. But it does suggest that the desire to get something published in Nature or Science or the New England Journal of Medicine can overcome qualms about carelessly generated and poorly curated data.
No one wants to go back to the slow pace of research that characterized the old order: lab researchers all love the ability to effortlessly access the literature, to order antibodies or molecular probes today and start an experiment tomorrow, to perform deep sequencing on a regular basis, generating Everest-sized mountains of data in minuscule periods. But none of these cause the current crisis of confidence.
I have had the sense over the past decade that the pace of research is picking up, driven by novel technology. I still hope and believe this to be true. But if nine out of ten “landmark” lab findings aren’t real (or three of four, if you are an optimist), then it isn’t the pace of research that has increased; it’s the volume of scientific sewage. And that is an important cultural issue for the world’s laboratories.