Fibrillar β-amyloid plaques are a histological hallmark of Alzheimer’s disease (AD) 1. Noninvasive β-amyloid detection by the recently approved PET brain imaging agents may improve accuracy in diagnosing cognitive impairment 2–5. [18F]Flutemetamol injection is a PET agent approved in 2013 in the USA 6 and in 2014 in the EU 7 for imaging of brain β-amyloid plaque density in patients with cognitive impairment after it was studied in 761 patients in nine clinical trials 8–12. The active component, [18F]flutemetamol, is a neutral thioflavin derivative of [11C]PiB with a high affinity (Kd: 6.7 nmol/l) for amyloid 13,14. Its significant cortical uptake differentiates probable AD patients from aged healthy volunteers 11,12.
Amyloid PET images present unique challenges to readers compared with other nuclear medicine images. Differentiation of nonspecific uptake in white matter and brainstem from specific binding to amyloid plaques in grey matter requires an understanding of brain anatomy and experience in PET image interpretation. Atrophic thinning of cortical grey matter presents a further complication. Historically, clinical development of PET tracers included in-person training of image reader 15,16. However, as these agents obtained marketing approval, regulatory bodies promoted the development and validation of scalable training programmes for image readers, such as self-guided, electronic media-based approaches 17,18. These are intended to standardize training and encourage adoption of an objective image interpretation methodology by the end users.
GE Healthcare developed a self-guided electronic training programme (ETP) to train nuclear medicine personnel to accurately and reproducibly orientate [by both nuclear medicine technologists (NMTs) and readers] and interpret (readers only) [18F]flutemetamol PET images for detecting significant brain amyloid uptake. ETP effectiveness was assessed by evaluating the performance (accuracy and reproducibility) of five readers who interpreted PET images from young and elderly patients with absent, early, probable or late dementia. The ETP was initially developed for use in the USA, but more recently, has been approved for use in both Europe and Japan 19,20.
Materials and methods
Description of the electronic training programme
The ETP (http://www.readvizamyl.com but preloaded on dedicated computers for the purpose of this study) familiarized nuclear medicine physicians, radiologists and technologists with the proper orientation and interpretation of [18F]flutemetamol PET images, as well as providing background information on brain anatomy and pathophysiology useful for image interpretation. The programme comprised four interactive modules including self-assessments covering the following:
Review of brain anatomy; introduction to brain distribution of [18F]flutemetamol using axial, sagittal and coronal views.
Image orientation and alignment; choice of colour scales for reading.
Strategies for classifying a scan as normal (negative) or abnormal (positive), evaluating five key regions using appropriate viewing planes, anatomic landmarks and recognizing atrophy (Fig. 1).
Independent sample images classified as positive (abnormal) or negative (normal) for the presence of amyloid.
To complete the testing process, the trainee was required to independently read and classify images and was provided expert feedback on a series of images. To be deemed ready to interpret [18F]flutemetamol images, users had then to correctly assign at least 14/15 images in the test at the end of the training.
The programme emphasized the importance and benefits of a colour (e.g. rainbow or Sokoloff) scale to facilitate identification of elevated levels of [18F]flutemetamol activity. By setting a known negative region, such as the cerebellar cortex, to 30%, and/or using pons activity to set near maximal intensity (90%), the colour gradient enabled differentiation between positive and negative regions of amyloid uptake in the cortex.
Electronic training programme validation
Training in this study consisted of image display and orientation (for NMTs and readers) and training nuclear medicine physicians or radiologists to interpret the images (Fig. 2).
Selection and training of nuclear medicine technologists and image readers
Eligible NMTs were certified, practicing in the USA and had predominantly routine clinical (nonresearch) experience in acquiring, processing and orienting nuclear brain images. Candidates were excluded if they were experienced in any type of amyloid imaging, had ever been debarred from clinical research or were unwilling to disclose potential conflicts of interest. Each candidate was required to watch the orientation module of the self-guided ETP, review the test images provided and orient them correctly in three different planes without assistance within three tries.
Each reader candidate was a board-certified nuclear medicine physician or a board-certified radiologist with clinical nuclear medicine training practicing in the USA, and extensive experience reading nuclear medicine images in a clinical, nonresearch setting. Exclusion criteria were the same as those for the NMTs. After completing the interactive content of the ETP, each reader candidate had to answer the review questions and read the test images without assistance. The reader candidate was required to pass the training programme to participate in the study.
All ETP training, orientation and reading was conducted at a site operated by a contract research organization, the American College of Radiology Image Metrix in Philadelphia, Pennsylvania, USA.
Imaging study participants and ethics
Images (n=276) collected from nine previous clinical studies of [18F]flutemetamol injection (Table 1) had been acquired from patients across the cognitive spectrum. These included young (age≤40 years) and elderly (age≥55 years) healthy volunteers; patients with amnesic mild cognitive impairment (aMCI) or probable AD; patients with known/suspected normal pressure hydrocephalus; and end-of-life patients (Table 1).
This study was carried out according to the Declaration of Helsinki 21, the International Conference on Harmonisation Good Clinical Practice guideline 22 and all applicable laws and regulations. Independent Ethics Committee (IEC) or Institutional Review Board (IRB) approval of the protocol was not sought because no patients received an investigational product. Patients whose brain images were used had participated in IRB/IEC-approved studies and they (or their legally authorized representative) had consented in writing to participate; data usage was considered to be covered by the previous consent. This study was registered on http://www.Clinicaltrials.gov (NCT01672827) and was carried out between 9 July 2012 (start of NMT training) and 23 August 2012 (database lock).
Image acquisition and selection
Image acquisition started ∼90 min after the administration of [18F]flutemetamol injection (VIZAMYL™; GE Healthcare, Little Chalfont, UK) and comprised six 5 min frames after a 185 MBq dose (n=208) or five 2 min frames after a 370 MBq dose (n=68). To ensure similar image signals, only the first four imaging frames (20 min) were used from 185 MBq scans to approximate the image signal of a 10 min, 370 MBq scan.
Depending on the contributing study, image selection was complete (where histopathological SoT was available) or from a random sample (in larger studies without a SoT). No images selected for the blinded image evaluation portion of the validation study had been used previously in the reader training programme. To assess intrareader reproducibility (IRR), 29 (10.5%) of the 276 images were selected at random and duplicated, and the originals and duplicates were included in the image set, for a total of 305 images.
Conduct of the blinded read
The set of 305 images (Table 2; populations 4 and 5) was divided into five 61-image sets; each was distributed to one of five ETP-trained NMTs. Each NMT checked (and corrected as needed) the orientation of the images in his/her set. The recombined image sets were then randomized for reading without anatomic images. According to routine practice by many US clinical sites, if a reader wanted reorientation of an image during the blinded image read, the reader could reorient the image himself/herself or request that it be done by an on-call ETP-trained NMT. While blinded to all patient clinical information, each reader interpreted each patient’s images, classifying the patient as normal (negative) or abnormal (positive) for the presence of amyloid and concluded the interpretation before moving to the next image set; no images could be excluded or classified as unevaluable. After all images had been read without anatomic images, an identical second reading session was performed, for which a subset of the [18F]flutemetamol images (n=149) had been rerandomized for reading with associated anatomic images (MRI or CT). The readers had no contact with any GE Healthcare employees before or during the read and the ETP was their only source of training information.
Pathological standard of truth measures
Postmortem histopathological standards of truth were determined for 68 patients on the basis of two blocks per each of eight brain regions (midfrontal lobe, superior temporal, middle temporal, inferior parietal, anterior cingulate, posterior cingulate, precuneus and primary visual cortex) as outlined in Curtis et al. 16. Briefly, for each block, three slides were prepared, stained with Bielschowsky silver stain and read independently by two blinded neuropathologists who analysed five random fields of view and reached consensus on neuritic plaque density; these were later converted into a plaque abundance score (modified CERAD criteria 1,16) from 0 (none) to 3 (frequent). The regional scores were derived from the mean of all slide scores, which in turn were the mean of all field of view scores. Ultimately, each regional score was dichotomized to a ‘negative’ (≤1.5) or a ‘positive’ (>1.5) amyloid pathology outcome, and if at least one region was positive, the patient was declared positive for amyloid pathology.
Biopsy-based standards of truth were obtained from 36 patients suspected to have normal pressure hydrocephalus 8–10; five fields of view for each of up to three slides of a single fixed Bielschowsky silver stain-stained tissue section per patient were examined and scored as detailed above. Averaged scores were dichotomized as detailed above.
Statistical analysis plan
All statistical analyses were carried out using statistical analysis software (SAS Institute Inc., Cary, North Carolina, USA). Seven analysis populations were defined for this study (Table 2). Each PET image classification was compared with the corresponding SoT results, where available, and categorized as true positive (TP), true negative (TN), false positive (FP) or false negative (FN). These were counted (n) and used to calculate validity measures: sensitivity=nTP/(nTP+nFN); specificity=nTN/(nTN+nFP). Two-sided exact 95% binomial confidence intervals were calculated for sensitivity and specificity for each reader and the majority interpretation (the image interpretation made independently by at least three of the five readers). The null hypothesis stated that at least one of the lower bounds of the two-sided 95% confidence interval was 70% or more; the alternative hypothesis stated that neither was. A sample size of 39 abnormal and 50 normal patients was calculated to provide ∼90% power, assuming a true sensitivity of 92% or more and a true specificity of 90% or more. IRR was measured by percent agreement. Inter-reader agreement (IRA) was analysed using Fleiss’ κ statistics. κ values were classified as follows: 0.6–0.7=good; >0.7–0.8=very good; and >0.8=excellent 23,24.
Of six NMT candidates screened, the first five were tested and passed the orientation module of the training programme. Two NMTs were on-call during the blinded visual image interpretation sessions in case a reader requested reorientation of an image. Of 18 reader candidates screened, the first five were trained with the ETP and tested. Each of the five trainees (three nuclear medicine physicians and two radiologists) correctly scored all 15 test images at the end of the ETP and subsequently participated in the study.
The overall reader performance resulting from the primary study objective (sensitivity and specificity without anatomic images for population 1, Table 3) was high, with sensitivity ranging from 84 to 94% (median 92%, majority 94%) and specificity ranging from 77 to 96% (median 81%, majority 92%). Reading accuracy with anatomic images for population 1 was also strong, with sensitivity ranging from 86 to 98% (median 92%, majority 94%) and specificity ranging from 80 to 94% (median 85%, majority 94%). Similar performance was found in the reading accuracy for populations 2 and 3 (Table 3).
When evaluating images from analysis population 4, all readers except one (reader 4) achieved excellent (>0.8) κ scores. When the subset of patients with aMCI was evaluated (analysis population 6), IRA was excellent.
IRR for analysis population 5 was similarly excellent: 100% for reader 1, 97% for readers 2 and 4 and 93% for readers 3 and 5. For aMCI patients (n=8, analysis population 7), IRR was 100% for all five readers.
In populations 1–3, readers interpreted up to 16% of images as FPs and up to 9% as FNs, with or without anatomic image support (see bold and italic values in Table 4 for population 1 data).
FPs are attributed to equivocal or low neuritic plaque levels with a substantial number of diffuse plaques; two autopsied cases had a low neuritic plaque score, but heavy diffuse plaque loads that may have elevated the [18F]flutemetamol PET signal to clearly abnormal levels. In a case from the biopsy cohort, underestimation of the global amount of amyloid may have occurred during the histopathological assessment because of the small fraction of brain tissue analysed. No FP interpretations were found by majority assessment in the healthy volunteers. FN interpretations are considered to be related to focal atrophy, a borderline neuritic plaque density or both; anatomic images did not resolve the FN. Finally, discordant interpretations may have derived from motion artefacts and a combination of the latter and borderline pathology.
The need for training of nuclear medicine professionals in general molecular imaging was identified in a recent joint report by the European Association of Nuclear Medicine and the Society of Nuclear Medicine and Molecular Imaging 25, as well as in past reports by other nuclear medicine bodies 26,27. Although digital training resources have been created for radiology 28 and nuclear medicine 29–31, to date, only two such programmes for molecular imaging have been described 32,33. With the development of new molecular imaging agents, there is a parallel need for agent-specific training rapidly deployable globally, and web-based programmes may help fulfil this need. We therefore developed an ETP providing background anatomy and pathophysiology needed to interpret [18F]flutemetamol images. Originally DVD based, the ETP is now on the web 19.
This study showed that the self-guided ETP effectively trains readers inexperienced in assessing brain amyloid to accurately and reproducibly interpret [18F]flutemetamol PET images. Sensitivity and specificity were high for all readers. The availability of anatomic images (CT/MRI) did not significantly alter results and the results were similar across the analysis populations. IRA and IRR overall were very good or excellent.
The high image interpretation accuracy that we observed after ETP training is similar to that found after in-person training for the 68 autopsy patients 16; in both cases, majority reads were more than 86% sensitive and more than 84% specific, suggesting effective training in image interpretation. It remains important to maintain an inquisitive eye towards equivocal scans: poststudy discussions with readers suggest that stringent adherence to the ETP criteria is key to the accurate visual interpretation of [18F]flutemetamol images. Similar to when in-person reader training was used in an earlier study 16, with the ETP, the causes of false image interpretations appear to be cortical atrophy, head motion during scans and predominance of high-frequency diffuse amyloid plaque 25. FN reads because of cortical atrophy-related difficulty localizing grey matter can be reduced by examining the striatum or inferior parietal regions, which are less subject to atrophy 20. Motion artefacts can be reduced by using appropriate head supports and flexible head restraints. Finally, diffuse plaques may result in FP amyloid pathology if neuritic plaque density alone is used as the standard of truth 34. However, the most recent criteria for the histopathological diagnosis of AD have been updated to include the pathological detection of both neuritic and diffuse plaques 35, in contrast with past official guidelines 1.
The number of readers (five) was chosen on the basis of precedents set in previous clinical trials of [18F]flutemetamol 11 and other amyloid PET imaging agents 15,36,37, and on agreement with regulatory bodies, but nevertheless, may not represent all readers. Bias was reduced as much as possible by blinding readers to patient clinical information and having images read first without access to anatomic images. The patients in this study may not fully reflect those who will undergo amyloid PET scanning in clinical practice, for whom the decision to image will likely be made on the basis of criteria such as the Alzheimer’s Association appropriate use criteria 38). Nevertheless, the accuracy of image interpretation for the varied populations examined in this study shows that the methodological robustness derived from the ETP enables a reliable assessment of amyloid pathology load even in the absence of pre-existing clinical information. Moreover, the study comprised a limited set of patients (n=68) for whom autopsy confirmation was available. Larger numbers would make the results more robust; however, no considerable differences in sensitivity or specificity were observed across groups with varying standards of truth.
This study validated the self-guided ETP for accurately and reproducibly interpreting [18F]flutemetamol PET images. Images included in the validation had been acquired from patients representing a broad spectrum of cognitive function and amyloid burden. After receiving the training, readers previously inexperienced in amyloid PET image interpretation performed well and showed high sensitivity, specificity, IRA and IRR, which are all standard metrics for assessing a diagnostic radiopharmaceutical. The availability of this electronic media-based training programme obviates the need for costly, cumbersome, in-person training. It also prevents potential variations in emphasis or depth of training across sessions, depending on the trainer or the audience, and thus represents a consistent and scalable approach to reader training.
The authors would like to acknowledge the writing assistance provided by Enrico R. Fantoni, PhD of GE Healthcare Amersham, UK, and Stacy Simpson Logan, CMPP, of Winfield Consulting.
This study was funded by GE Healthcare.
Conflicts of interest
Christopher Buckley, Paul Sherwin, Adrian Smith, Jan Wolber and Sharon Weick are employees of GE Healthcare. David Brooks is an ex-employee and consultant for GE Healthcare.
1. Mirra S, Heyman A, McKeel D, Sumi S. The consortium to establish a registry for Alzheimer’s disease (CERAD). Part II. Standardization of the neuropathologic assessment of Alzheimer’s disease. Neurology 1991; 46:142–145.
2. Schipke CG, Peters O, Heuser I, Grimmer T, Sabbagh MN, Sabri O, et al. Impact of beta-amyloid-specific florbetaben pet imaging on confidence in early diagnosis of Alzheimer’s disease. Dement Geriatr Cogn Disord 2012; 33:416–422.
3. Frederiksen KS, Hasselbalch SG, Hejl AM, Law I, Højgaard L, Waldemar G. Added diagnostic value of 11
C-PiB-PET in memory clinic patients with uncertain diagnosis. Dement Geriatr Cogn Dis Extra 2012; 2:610–621.
4. Cohen J, Zhong Y, Karlawish J. Impact of amyloid beta imaging on diagnosis and clinical management of hypothetical patients with mild cognitive impairment or dementia (P3.210). Neurology 2014; 82 (P3):210.
5. Beach TG, Schneider JA, Sue LI, Serrano G, Dugger BN, Monsell SE, et al. Theoretical impact of florbetapir (18
F) amyloid imaging on diagnosis of Alzheimer dementia and detection of preclinical cortical amyloid. J Neuropathol Exp Neurol 2014; 73:948–953.
8. Wong DF, Moghekar AR, Rigamonti D, Brašić JR, Rousset O, Willis W, et al. An in vivo evaluation of cerebral cortical amyloid with [18
F]flutemetamol using positron emission tomography compared with parietal biopsy samples in living normal pressure hydrocephalus patients. Mol Imaging Biol 2013; 15:230–237.
9. Leinonen V, Rinne JO, Virtanen KA, Eskola O, Rummukainen J, Huttunen J, et al. Positron emission tomography with [18
F]flutemetamol and [11
C]PiB for in vivo detection of cerebral cortical amyloid in normal pressure hydrocephalus patients. Eur J Neurol 2013; 20:1043–1052.
10. Wolk DA, Grachev ID, Buckley C, Kazi H, Grady MS, Trojanowski JQ, et al. Association between in vivo fluorine 18-labeled flutemetamol amyloid positron emission tomography imaging and in vivo cerebral cortical histopathology. Arch Neurol 2011; 68:1398–1403.
11. Vandenberghe R, van Laere K, Ivanoiu A, Salmon E, Bastin C, Triau E, et al. 18
F-flutemetamol amyloid imaging in Alzheimer disease and mild cognitive impairment: a phase 2 trial. Ann Neurol 2010; 68:319–329.
12. Nelissen N, van Laere K, Thurfjell L, Owenius R, Vandenbulcke M, Koole M, et al. Phase 1 study of the Pittsburgh compound B derivative 18
F-flutemetamol in healthy volunteers and patients with probable Alzheimer disease. J Nucl Med 2009; 50:1251–1259.
13. Luthra SK, Robins EG. Gouverneur V, Müller K. 18
F-labelled tracers for PET oncology and neurology applications. Fluorine in pharmaceutical and medicinal chemistry: from biophysical aspects to clinical applications. Hackensack, NJ: Imperial College Press; 2012. 383–441.
14. Juréus A, Swahn BM, Sandell J, Jeppsson F, Johnson AE, Johnström P, et al. Characterization of AZD4694, a novel fluorinated Aβ plaque neuroimaging PET radioligand. J Neurochem 2010; 114:784–794.
15. Clark CM, Pontecorvo MJ, Beach TG, Bedell BJ, Coleman RE, Doraiswamy PM, et al. Cerebral PET with florbetapir compared with neuropathology at autopsy for detection of neuritic amyloid-beta plaques: a prospective cohort study. Lancet Neurol 2012; 11:669–678.
16. Curtis C, Gamez JE, Singh U, Sadowsky CH, Villena T, Sabbagh MN, et al. Phase 3 trial of flutemetamol labeled with radioactive fluorine 18 imaging and neuritic plaque density. JAMA Neurol 2015; 72:287–294.
18. Yang L, Rieves D, Ganley C. Brain amyloid imaging – FDA approval of florbetapir F18 injection. N Engl J Med 2012; 367:885–887.
19. GE Healthcare. Vizamyl (flutemetamol F 18 injection) electronic reader training programme; 2016. Available at: https://www.readvizamyl.com/
. [Accessed 15 September 2016].
21. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013; 310:2191–2194.
22. International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use. ICH harmonised tripartite guideline. Yokohama, Japan: U.S. Department of Health and Human Services, Food and Drug Administration; 1996.
23. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76:378–382.
24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174.
25. Minoshima S, Drzezga AE, Barthel H, Bohnen N, Djekidel M, Lewis DH, et al. SNMMI procedure standard-EANM practice guideline for amyloid PET imaging of the brain. J Nucl Med 2016; 57:1316–1322.
26. Graham MM, Jacene HA. Molecular imaging training for nuclear medicine residents. J Nucl Med 2012; 53:655–657.
27. Guiberteau MJ, Graham MM. ACR-SNM Task Force on Nuclear Medicine Training: report of the task force. J Nucl Med 2011; 52:998–1002.
28. Hoa D, Micheau A, Gahide G. Creating an interactive web-based e-learning course: a practical introduction for radiologists. Radiographics 2006; 26:e25. quiz e25.
29. Diessl S, Verburg Fa, Hoernlein A, Schumann M, Luster M, Reiners C. Evaluation of an internet-based e-learning module to introduce nuclear medicine to medical students: a feasibility study. Nucl Med Commun 2010; 31:1063–1067.
30. Wallis JW, Miller MM, Miller TR, Vreeland TH. An internet-based nuclear medicine teaching file. J Nucl Med 1995; 36:1520–1527.
31. Wallis JW, Parker JA. Use of the Internet for teaching in nuclear medicine. Semin Nucl Med 1998; 28:165–176.
32. Bullich S, Catafau A, Senda M, Khodaverdi-Afaghi V, Stephens A. Performance of 18
F-florbetaben PET image reading training in Japanese language. J Nucl Med, 2016:571822.
33. Seibyl J, Catafau AM, Barthel H, Ishii K, Rowe CC, Leverenz JB, et al. Impact of training method on the robustness of the visual assessment of 18
F-florbetaben PET scans: results from a phase 3 trial. J Nucl Med 2016; 57:900–906.
34. Lockhart A, Lamb JR, Osredkar T, Sue LI, Joyce JN, Ye L, et al. PIB is a non-specific imaging marker of amyloid-beta (Aβ) peptide-related cerebral amyloidosis. Brain 2007; 130:2607–2615.
35. Hyman BT, Phelps CH, Beach TG, Bigio EH, Cairns NJ, Carrillo MC, et al. National Institute on Aging – Alzheimer’s Association guidelines for the neuropathologic assessment of Alzheimer’s disease. Alzheimers Dement 2012; 8:1–13.
36. Clark CM, Schneider JA, Bedell BJ, Beach TG, Bilker WB, Mintun MA, et al. Use of florbetapir-PET for imaging beta-amyloid pathology. JAMA 2011; 305:275–283.
37. Ong KT, Villemagne VL, Bahar-Fuchs A, Lamb F, Langdon N, Catafau AM, et al. Aβ imaging with 18
F-florbetaben in prodromal Alzheimer’s disease: a prospective outcome study. J Neurol Neurosurg Psychiatry 2014; 86:431–436.
38. Johnson KA, Minoshima S, Bohnen NI, Donohoe KJ, Foster NL, Herscovitch P, et al. Update on appropriate use criteria for amyloid PET imaging: dementia experts, mild cognitive impairment, and education. J Nucl Med 2013; 54:1011–1013.