Guy's Hospital, St. Thomas Street, London, United Kingdom.
Disclosure: The authors declare no conflicts of interest.
Address for correspondence: Martin Utley, PhD, Clinical Operational Research Unit, Department of Mathematics, University College London, 4 Taviton Street, London, WC1H 0BT, United Kingdom. E-mail: firstname.lastname@example.org
As discussed elsewhere in this supplement (see for instance Fiorentino and Treasure1), the evidence base concerning the practice of pulmonary metastasectomy consists mainly of surgical follow-up studies reporting clinicopathologic characteristics of patients and postmetastasectomy survival. It is in the nature of follow-up studies that the authors retrieve data on patients who have had the operation under review but are rarely in a position to report on the size of the population from which they were drawn. One exception in the case of pulmonary metastasectomy is the work of Wade et al.2 They reported on 76 patients who had pulmonary metastasectomy for colorectal cancer and, with access to registry data, were able to report that these 76 were among 22,715 patients who had undergone surgery for primary colorectal cancer, 2659 of whom had pulmonary metastases, with metastases considered to be restricted to the lung in 514.
This high degree of selection presents a fundamental problem when interpreting survival data (Figure 1). Cancer patients considered for surgery have, as a group, less extensive disease than those not considered surgical candidates. For this reason, observed or perceived differences in outcome between the two groups cannot be attributed solely to the surgery. Gene Blackstone referred this as “the treachery of work-up bias.”3
With particular reference to pulmonary metastasectomy, Torkel Aberg raised the question in 1980 as to whether the survival rates of approximately 30% at 5 years were truly attributable to the surgery or were in reality because of the careful selection of patients.4 The question remained unanswered 17 years later when he again warned caution in the interpretation of postmetastasectomy survival.5
RESPONSES TO IMPERFECT EVIDENCE
There are three distinct responses to imperfect evidence regarding the effectiveness of medical or surgical interventions. One response is to rely on one's clinical judgment to assess whether the nature of an intervention is likely to be effective given the accumulated knowledge and experience pertaining to the condition in question. An extreme example of where rigorous evaluation of clinical effectiveness is clearly not required is a “mother's kiss” to remove a bead from a child's nostril by blowing into its mouth.6 Cataract surgery is a more familiar surgical example where the benefit is immediately evident. From our own work, we would argue that a new, less burdensome way of achieving the surgical objectives of aortic root replacement in Marfan syndrome has self-evident benefits.7 Whether the putative mechanism whereby pulmonary metastasectomy offers survival benefit is consistent with current understanding of the dissemination of colorectal cancer is discussed elsewhere.8
Another response is to call for better evidence. It could be argued that, whenever faced with the possibility of selection bias, the only way to measure the benefit of surgery is to conduct a randomized trial. However, randomized controlled trials are expensive, demanding, can be divisive, and are particularly challenging when the surgery in question is in existing and widespread practice.9
Third, a complementary approach is to use modeling to get as much information as possible from the available data. This may involve working with survival data pertaining to operated and unoperated cases to get an upper estimate of any benefit conferred by surgery to weigh against surgical morbidity.10 Modeling has also been used to predict how patients undergoing a trial intervention would have progressed had they received standard treatment; the work of Jackson and Aspden11 related to acute myeloid leukemia is a prime example.
CONSTRUCTING A COMPARATOR GROUP
Although most authors are careful to point out that pulmonary metastasectomy patients are a carefully selected group, implicit comparisons are made between observed postmetastasectomy survival and survival observed among other patients with metastatic colorectal cancer. Such comparison with a completely unselected group of patients, known to have disseminated disease at diagnosis of the primary, is not warranted.
As part of a program of research to establish whether there is a sufficiently strong case to be made for conducting a randomized controlled trial of pulmonary metastasectomy in colorectal cancer, we undertook a simple modeling study, the details of which are presented elsewhere.12 The aim of the study was to use the extensive resource of a cancer registry to construct some form of comparator group against which to assess the postmetastasectomy survival data reported in the literature.
In doing so, we wanted to account for differences between metastasectomy patients and the broader patient population with colorectal cancer. We focused on two key, linked differences between the two groups of patients. First, although some pulmonary metastasectomy patients have disseminated disease at the time that the primary colorectal cancer is diagnosed, they predominantly have less extensive disease13 and so, arguably, represent more indolent tumors. In addition, there is typically an interval of 2 to 3 years between the resection of the primary colorectal cancer and pulmonary metastasectomy, referred to as the disease-free interval (DFI). Again, this may be indicative of tumor indolence, and there is the possibility that this selects those patients with a natural course of longer survival. These two factors were chosen as they were available in the registry and in a number of case series reports.
We built a model to predict survival among a group of patients matched for Dukes stage with a cohort of pulmonary metastasectomy patients. Importantly, we predicted survival beyond an interval following diagnosis (termed here the “death-free interval”) with this interval analogous to and chosen to be consistent with the “DFI” reported for the metastasectomy patients. To populate this conditional survival model, we used raw survival data from the Thames Cancer Registry in the UK that receives notification of cancer diagnoses and subsequent deaths for a population of approximately 12 million people. Example output of this model, using the stage-mix and “DFI” reported by McCormack et al.,14 can be seen in Figure 2.
Considerable caution is required in interpreting the output of the model described above. In particular, the fact that the modeled, comparator survival curve lies within the 95% confidence interval of the observed 5-year survival rate reported for metastasectomy should not be interpreted as evidence that pulmonary metastasectomy is ineffective (see Ref. 12 for a full discussion of the limitations). That said, the findings of the simple modeling exercise undermine the notion that the reported 5-year survival rates of 30 to 40% after pulmonary metastasectomy in colorectal cancer are, of themselves, sufficient proof that survival benefit is conferred to patients by this operation. Used in this way, as one form of evidence among many, modeling has a useful role to play in evaluating and interpreting non-trial data.
1. Fiorentino F, Treasure T. A plea for consistency in the reporting of surgical series: an analysis of 51 follow-up reports of pulmonary metastasectomy in colorectal carcinoma. J Thorac Oncol
2. Wade TP, Virgo KS, Li MJ, et al. Outcomes after detection of metastatic carcinoma of the colon and rectum in a national hospital system. J Am Coll Surg
3. Blackstone EH, Lauer MS. Caveat emptor: the treachery of work-up bias. J Thorac Cardiovasc Surg
4. Aberg T, Malmberg KA, Nilsson B, et al. The effect of metastasectomy: fact or fiction? Ann Thorac Surg
5. Aberg T. Selection mechanisms as major determinants of survival after pulmonary metastasectomy. Ann Thorac Surg
6. Glasziou P, Chalmers I, Rawlins M, et al. When are randomised trials unnecessary? Picking signal from noise. BMJ
7. Pepper JR, Golesworthy T, Utley M, et al. Manufacturing and placing a bespoke support for the Marfan aortic root: description of the method and technical results and status at one year for the first ten patients. Interact Cardiovasc Thorac Surg
8. Treasure T, Utley M, Hunt I. When professional opinion is not enough: surgical resection of pulmonary metastases. BMJ
9. Treasure T. Are randomised trials needed in the era of rapidly evolving technologies? Eur J Cardiothorac Surg
10. Utley M, Fiorentino F, Treasure T. Obtaining an upper estimate of the survival benefit associated with radical surgery for mesothelioma. Eur J Cardiothorac Surg
11. Jackson R, Aspden P. Treatment evaluation—a modelling approach with application to acute myeloid leukaemia. J Oper Res Soc
12. Utley M, Treasure T, Linklater K, et al. Better out than in? The resection of pulmonary metastases from colorectal tumours. In Operations Research for Health Care Delivery Engineering. Proceedings of 33rd International conference on Operational Research Applied to Health Services (ORAHS 2007), Saint Etienne, France, 2008. Pp. 493–500.
13. Fiorentino F, Hunt I, Teoh K, et al. Pulmonary metastasectomy in colorectal cancer: a systematic review and quantitative synthesis. J R Soc Med
14. McCormack PM, Burt ME, Bains MS, et al. Lung resection for colorectal metastases. 10-year results. Arch Surg