The aim of this study was to evaluate the accuracy of a computer-based system for the automated diagnosis of melanoma in the hands of nonexpert physicians. We performed a prospective comparison between nonexperts using computer assistance and experts without assistance in the setting of a tertiary referral center at a University hospital. Between February and November 2004 we enrolled 511 consecutive patients. Each patient was examined by two nonexpert physicians with low to moderate diagnostic skills who were allowed to use a neural network-based diagnostic system at their own discretion. Every patient was also examined by an expert dermatologist using standard dermatoscopy equipment. The nonexpert physicians used the automatic diagnostic system in 3827 pigmented skin lesions. In their hands, the system achieved a sensitivity of 72% and a specificity of 82%. The sensitivity was significantly lower than that of the expert physician (72 vs. 96%, P = 0.001), whereas the specificity was significantly higher (82 vs. 72%, P<0.01). Three melanomas were missed because the physicians who operated the system did not choose them for examination. The system as a stand-alone device had an average discriminatory power of 0.87, as measured by the area under the receiver operating characteristic curve, with optimal sensitivities and specificities of 75 and 84%, respectively. The diagnostic accuracy achieved in this clinical trial was lower than that achieved in a previous experimental trial of the same system. In total, the performance of a decision-support system for melanoma diagnosis under real-life conditions is lower than that expected from experimental data and depends upon the physicians who are using the system.
aDepartment of Software Engineering, Upper Austria University of Applied Sciences, Hagenberg
bDepartment of Biomedical Engineering, University of Health Sciences, Medical Informatics and Technology, Hall
cDepartment of Dermatology, University of Vienna Medical School, Vienna, Austria
Correspondence to Dr Harald Kittler, MD, Medical University of Vienna, Waehringer Guertel 18-20, Vienna, Austria
Tel: +43 140400 7711; fax: +43 1408 1287 2099;
Preliminary results of this study were presented at the AMIA Meeting 2007 in Chicago and were published in AMIA Annu Symp Proc 2007:191–195 under the title ‘Applying a decision-support system in clinical practice: results from melanoma diagnosis'.
Received 3 November 2008 Accepted 5 February 2009