Share this article on:

Identifying Potential Adverse Effects by Patients’ Ratings: A Proof-of-Concept Study of a Novel Approach

Mago, Rajnish MD*; Kloos, Angelica DO; Daskalakis, Constantine ScD; Shwarz, Michelle MSEd*; Rovner, Barry W. MD

Journal of Clinical Psychopharmacology: December 2012 - Volume 32 - Issue 6 - p 828–831
doi: 10.1097/JCP.0b013e318270e2e2
Brief Reports

Objective Methods to evaluate adverse effects of medications are significantly underdeveloped compared to those for efficacy. In this pilot proof-of-concept study, we preliminarily compared a novel approach—the Symptom Assessment Tool (SAT)—to a systematic and detailed assessment by a physician for identifying symptoms that were potentially adverse effects (sensitivity) and excluding symptoms that were unlikely to be adverse effects (specificity).

Methods A symptom inventory and rating of symptom severity were completed before starting a psychotropic medication (or increasing its dose), and again 2 weeks later. Each symptom was systematically assessed by the patient-rated SAT and by a physician and was classified as either a potential or unlikely adverse effect. The primary analysis compared the classification of symptoms by the SAT to that by the physician. Potential adverse effects were also subcategorized as possible or probable adverse effects.

Results A sample of 193 symptoms from 15 adults was evaluated, only 37.3% of which were considered potential adverse effects by the physician. Sensitivity of the SAT compared to physician’s assessment was 90.3% for potential adverse effects and 97.5% for the subgroup of probable adverse effects. The SAT correctly identified 63.6% of the symptoms as unlikely adverse effects (specificity), and its negative predictive value was 91.7%.

Conclusions The SAT, appropriate for its intended use as a screening tool, had high sensitivity and moderate specificity and could present physicians with a limited number of potential adverse effects for further assessment and intervention. Further evaluation and refinement of this approach is warranted.

From the *Department of Psychiatry and Human Behavior, Jefferson Medical College, Philadelphia, PA; †Department of Child and Adolescent Psychiatry, Children’s National Medical Center, Washington, DC; ‡Division of Biostatistics, Thomas Jefferson University; and §Department of Neurology, Jefferson Medical College, Philadelphia, PA.

Received September 15, 2011; accepted after revision April 11, 2012.

Reprints: Rajnish Mago, MD, Department of Psychiatry and Human Behavior, Jefferson Medical College, 833 S Chestnut St, Suite 210 E, Philadelphia, PA 19107 (e-mail

This study was supported in part by a NARSAD Young Investigator Award to Dr Mago.

This paper was presented as a scientific and clinical report at the annual meeting of the American Psychiatric Association, May 2010, New Orleans, LA.

Adverse effects of psychotropic medications have a significant impact on the success of treatment. Symptoms believed to be adverse effects are cited as the most frequent reason for antidepressant discontinuation in clinical trials.1 Adverse effects are the reason most frequently cited by psychiatrists for choosing a particular antidepressant.2 Therefore, accurate information about the adverse effect profiles of different medications is essential. However, identification and assessment of adverse effects remains unsatisfactory both in clinical practice and in clinical trials. Patient- or clinician-rated scales for adverse effects are rarely used in clinical practice and clinical trials (except scales for specific “objective” adverse effects like extrapyramidal symptoms). For example, of 73 antidepressant clinical trials, only 14% reported use of any adverse effect rating scale, only 18% described assessing severity of the adverse effects, and only 12% reported assessing persistence of the adverse effects.3 Importantly, no study reported assessing the relationship of symptoms to the medication other than that they were reported as occurring during the trial.

The methodology for evaluation of adverse effects of medications is significantly underdeveloped compared to the methodology for assessment of efficacy.4 Each of the methods for identifying adverse effects has a variety of problems markedly limiting their use. Patients often do not spontaneously report symptoms that may be adverse effects to their physicians. Of patients who had adverse effects of their antidepressant, 45.8% reported that they had not discussed these with their physician or pharmacist.5 Open-ended questioning often misses symptoms that may be adverse effects. In a study using the Systematic Assessment for Treatment-Emergent Events,6 as many as 46% of even the symptoms that led to medical actions (beyond increased surveillance) were not identified on open-ended questioning including, surprisingly, 61% of events rated as severe and 65% of events causing severe functional impairment.7 However, amelioration of certain adverse effects may significantly improve a patient’s quality of life even if the patient was not spontaneously complaining about that adverse effect.8 Patient-rated symptom inventories, for example, Patient-Rated Inventory of Side Effects9 mainly ask patients to endorse symptoms present from a list. Therefore, they often yield a large number of symptoms, and the causal relationship with the medication is uncertain. Clinician-rated scales for adverse effects are often time consuming and impractical for clinical use.10 They sometimes require 30 to 60 minutes to complete,11,12 discouraging their use even in research settings. Most importantly, they identify symptoms, not adverse effects, that is, they do not include any standardized method to assess how likely it is that the symptoms present are, in fact, related to the medication.

The most important problem in assessing adverse effects is establishing the causal relationship between the putative adverse effect and the medication.13 Clinical researchers use the terms adverse event or adverse effect to mean “any unfavorable event that occurs during treatment regardless of cause.”14 However, of symptoms that were first detected by 3 approaches—open-ended questions, a focused interview specific to the medication being used, or a general review of body systems—only 38%, 44%, and 23%, respectively, were attributed to the medication by physicians.14 Thus, without assessment of causality, most symptoms would erroneously be thought to be adverse effects. In summary, current methods of eliciting symptoms that may be adverse effects fail to identify many adverse effects, and many symptoms identified are probably not adverse effects. In other words, both the sensitivity and specificity of these methods are low. An ideal method of identifying adverse effects would include the following: (a) identification of all symptoms present, (b) assessment of the causal relationship between the medication and each symptom, and (c) rating of the severity of these adverse effects so that their clinical significance can be appreciated and their management prioritized. We therefore developed the Symptom Assessment Tool (SAT) by integrating these 3 components and evaluated its potential validity and use in this pilot, proof-of-concept study.

Back to Top | Article Outline


Patients with a diagnosis of a depressive disorder or anxiety disorder based on clinical interview who were initiating treatment with, or increasing the dose of, any antidepressant, anxiolytic, or hypnotic medication were recruited from the outpatient practices of Thomas Jefferson University (Philadelphia, Pa). All other medications except the target medication that was being started (or whose dose was being increased) were required to have been at a stable dose for at least 4 weeks before initiating the study. Subjects were excluded if they had significant cognitive impairment, substance abuse or dependence in the past 30 days, or were expected to start any other medication during the study.

Back to Top | Article Outline

Description of the Symptom Assessment Tool

The tool consists of 4 components: (1) An inventory of the presence of a variety of symptoms that may be adverse effects of psychotropic medications. This list of symptoms is taken from the Systematic Assessment for Treatment-Emergent Events—Specific Inquiry,8 which has been used as a clinician-rated instrument. A broad range of symptoms that may be adverse effects of psychotropic medications is covered and is organized by organ systems for convenience. (2) Patient rating of the severity of each symptom present. (3) Comparison of the presence and severity of each symptom before and after starting the medication (or increasing its dose). Based on this, the SAT classifies the symptoms into 4 categories: “new” symptoms—not present before starting the medication; “same” symptoms—present before starting the medication and severity rating had not changed since then; “worse” symptoms—present before starting the medication but severity rating had worsened after starting the medication; and “better” symptoms—present before starting the medication and severity rating had improved since starting the medication. (4) For each symptom, patients answer a series of questions that either support or argue against the symptom being an adverse effect of the medication.

The third and fourth components described earlier are being referred to as the “causal algorithm,” intended to assess the likelihood that each symptom is in fact an adverse effect of the medication. Scoring on the causal algorithm is as follows: “new” or “worse” symptom, +1; “same” or “better” symptom, zero; data that support the association between the symptom and the medication (symptom worse when dose was increased or symptom better when dose decreased or subject did not take the medication or ever had the symptom with a similar medication in the past, or symptom worse within a few hours of taking medication) +1; data that support an alternative explanation for the symptom (symptom started or got worse within a week of starting another medication or symptom started or got worse within a week of becoming sick in any other way, or any other reason for the symptom) −1. The total score classifies each symptom as a potential adverse effect (score +1 or greater) or unlikely adverse effect (score zero or less). For secondary analysis, potential adverse effects were subdivided into symptoms that were possible adverse effects (score +1) or probable adverse effects (score +2 or greater).

Back to Top | Article Outline

Study Procedures

Visit 1

Before starting the target medication or increasing its dose, patients completed the inventory of symptoms and rated the severity of each symptom present.

Back to Top | Article Outline

Visit 2

Patients returned 14 ± 2 days after starting the target medication or increasing its dose. First, they were asked a standardized open-ended question by a research assistant: “Have you had any symptoms that you think may be side effects of the medication since you started taking it?” Patients then completed the same inventory of symptoms that they had completed at visit 1 and rated the severity of each symptom present. From symptoms present at visit 2, that is, after starting the medication, a random sample of up to 15 symptoms per patient were independently evaluated by both the SAT and a physician to assess the likelihood that they were adverse effects.

The physician’s assessment was conducted after the SAT evaluation and was blinded to patients’ responses on the symptom inventory, open-ended question, and SAT causal algorithm. First, the physician rated each symptom on the items of the Adverse Drug Reaction Probability Scale,13 the total score on which was classified each symptom into unlikely, possible, or probable adverse effects. For each symptom, the physician then asked additional questions as indicated and provided a final rating of each symptom as a possible or a probable adverse effect. On both the Adverse Drug Reaction Probability Scale (ADRPS) and the physician’s ratings, possible and probable adverse effects were combined and called potential adverse effects for the planned primary analysis. Whereas it is impossible to be certain which symptoms are truly adverse effects, the physician’s detailed interview including both structured questioning using the ADRPS (which has been widely used as an instrument to assess the relationship between a symptom and a medication) and additional questioning is the best possible reference standard for how likely it is that each symptom is an adverse effect of the medication. In addition, even this comprehensive assessment does not consider the symptom to definitely be an adverse effect but only to be potentially (possibly or probably) an adverse effect, that is, meriting further assessment, monitoring, and management.

Back to Top | Article Outline

Statistical Analysis

The unit of analysis used was the symptoms identified, rather than the subjects, and all symptoms were considered equivalent for the purpose of analysis. In the planned primary analysis, the physician’s final rating regarding each symptom was used as the reference standard for computing the test characteristics of SAT. Since multiple symptoms per patient were evaluated, observations from the same patient would be correlated to some extent and would be clustered within the patient. Therefore, for estimating the confidence intervals for sensitivity and specificity, we used logistic regression with the robust variances (standard errors) to take this clustering into account. Stata 11.0 was used for these analyses.

Back to Top | Article Outline


Fifteen patients with a median age of 35 years (range, 23–57 years), 8 men, 9 whites, and 9 with more than a high school education who completed both visits (ie, were evaluable) constitute the sample. Both the SAT and the physicians’ assessments were available for 193 symptoms. As shown in Tables 1 and 2, the SAT identified as potential adverse effects 90.3% (65/72) of symptoms considered potential adverse effects by the physician (the reference standard). Thus, its sensitivity for identification of symptoms that could be adverse effects (ie, potential adverse effects) was 90.3%. The SAT classified as unlikely adverse effects 63.6% (77/121) of symptoms considered unlikely adverse effects by the physician, that is, specificity was 63.6% (Tables 1 and 2). Of symptoms considered potential adverse effects by the SAT, subsequent physician’s final rating considered 59.6% to be potential adverse effects, that is, the positive predictive value of SAT was 59.6%. More importantly, of symptoms considered unlikely adverse effects by the SAT, 91.7% were rated by the physician as unlikely adverse effects, ie, the negative predictive value of the SAT was 91.7%. The corresponding test characteristics when the SAT was compared to the ADRPS rating are shown in Table 2.





Based only on the presence and severity rating on the symptom inventory, 59.1% (114/193) of the symptoms did not seem to be new or worse since baseline (Table 1). Surprisingly, 28.9% of these symptoms were considered potential adverse effects after methodical and comprehensive assessment by the physician (Table 1). The severity rating was into broad categories (eg, mild, moderate, and severe) that are hard to define and to rate reliably. Second, symptoms rated by patients as “mild,” “moderate,” or “severe” both before and after starting the medication may nevertheless be worse after starting the medication. Lastly, the character of the symptom sometimes changes. Approximately half of the symptoms that were “treatment-emergent” (ie, either new or worse) were considered unlikely adverse effects by the physician. The most common reason for this was having a clear alternative explanation for the symptom. Reasons for these discrepancies will be explored in subsequent studies.

In secondary analysis, sensitivity of the SAT was evaluated for the subgroup of symptoms that the physician considered to be probable adverse effects as opposed to just potential adverse effects (Table 3). Of 40 symptoms considered probable adverse effects by the physician’s rating, the SAT identified 39 as potential adverse effects, that is, sensitivity was 97.5%, and only one symptom was a false negative.



In response to the standardized open-ended question at the start of visit 2, patients reported a total of 32 symptoms (Table 1). Of these, 29 were also identified by the SAT symptom inventory. However, the open-ended question did not identify 66.7% (48/72) of symptoms that were identified on the symptom inventory and were rated by the physician as potential adverse effects, that is, its sensitivity was only 33.3%. Interestingly, of symptoms that were identified using the open-ended question, 82.8% were considered potential adverse effects by subsequent physician rating, that is, symptoms identified on open-ended questioning had a high positive predictive value.

Of the 193 symptoms identified by the symptom inventory and randomly selected for evaluation, the open ended question identified only 15.0% (29/193) and only 37.3% (72/193) were considered potential adverse effects by the physician (Table 1). Of the 11 body systems into which the symptom inventory was organized, the proportion of symptoms that were considered potential adverse effects by the physician varied from 0% to 54%, the highest being “Head” (eg, headaches, dizziness, etc) and “Genitourinary/Sexual.”

Back to Top | Article Outline


The high sensitivity and moderate specificity of the SAT found in this study are appropriate for its intended use—not as the final arbiter of whether a symptom is or is not an adverse effect but as a screening tool to precede physician assessment. It aims to identify all or almost all symptoms that could be adverse effects, many of which are likely to be missed if only spontaneous reporting or open-ended questioning are relied on, that is, high sensitivity or very few false negatives is its most important characteristic. In this study, the open-ended question missed two thirds of the potential adverse effects. Second, if patients complete the SAT before seeing their physicians, physicians will be presented with fewer symptoms, making it feasible to further assess and monitor them. In this study, the SAT ruled out approximately 40% symptoms from further consideration. In addition, symptoms considered probable adverse effects might be prioritized for assessment and management. The SAT will also provide physicians with additional data about each symptom to aid ongoing assessment by them.

In further development, some of the limitations of the present study will be addressed. Because this was a small pilot study, characteristics of the SAT will be evaluated in a larger sample of patients. Test-retest reliability of patient and physician ratings will be evaluated. In addition, data on the time taken to complete the SAT and on patient acceptability of the SAT assessment will be obtained.

A basic standardized approach to assessment of adverse effects is long overdue. The SAT may facilitate earlier and more complete identification of adverse effects by physicians, potentially leading to more active management. It will also help to identify which symptoms are probably not adverse effects, potentially reducing treatment nonadherence. In addition, it will facilitate research into a variety of questions related to adverse effects and their assessment. It may allow more accurate estimates of the prevalence of various adverse effects with medications and meaningful comparisons between medications.

Back to Top | Article Outline


The authors declare no conflicts of interest.

Back to Top | Article Outline


1. Roose SP. Compliance: the impact of adverse events and tolerability on the physician’s treatment decisions. Eur Neuropsychopharmacol. 2003; 13 (suppl 3): S85–S92.
2. Zimmerman M, Posternak M, Friedman M, et al.. Which factors influence psychiatrists’ selection of antidepressants? Am J Psychiatry. 2004; 161: 1285–1289.
3. Woodmansee C, Mago R, Shah S, et al.. Identification and Assessment of Adverse Effects in Antidepressant Clinical Trials. San Diego, CA: American Psychiatric Association; 2007.
4. Greenhill LL, Vitiello B, Riddle MA, et al.. Review of safety assessment methods used in pediatric psychopharmacology. J Am Acad Child Adolesc Psychiatry. 2003; 42: 627–633.
5. Bull SA, Hu XH, Hunkeler EM, et al.. Discontinuation of use and switching of antidepressants: influence of patient-physician communication. JAMA. 2002; 288: 1403–1409.
6. Levine J, Schooler NR. SAFTEE: a technique for the systematic assessment of adverse effects in clinical trials. Psychopharm Bull. 1986; 22: 343–381.
7. Levine J, Schooler NR. General versus specific inquiry with SAFTEE. J Clin Psychopharmacol. 1992; 12: 448.
8. Lampela P, Hartikainen S, Sulkava R, et al.. Adverse drug effects in elderly people—a disparity between clinical examination and adverse effects self-reported by the patient. Eur J Clin Pharmacol. 2007; 63: 509–515.
9. Rush AJ, Fava M, Wisniewski SR, et al.. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004; 25: 119–142.
10. Lindstrom E, Lewander T, Malm U, et al.. Patient-rated versus clinician-rated adverse effects of drug treatment in schizophrenia. Clinical validation of a self-rating version of the UKU Adverse Effect Rating Scale (UKU-SERS-Pat). Nord J Psychiatry. 2001; 55 (suppl 44): 5–69.
11. Lingjaerde O, Ahlfors UG, Bech P, et al.. The UKU adverse effect rating scale. A new comprehensive rating scale for psychotropic drugs and a cross-sectional study of adverse effects in neuroleptic-treated patients. Acta Psychiatr Scand Suppl. 1987; 334: 1–100.
12. Day JC, Wood G, Dewey M, et al.. A self-rating scale for measuring neuroleptic adverse-effects. Validation in a group of schizophrenic patients. Br J Psychiatry. 1995; 166: 650–653.
13. Naranjo CA, Busto U, Sellers EM, et al.. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981; 30: 239–245.
14. Greenhill LL, Vitiello B, Fisher P, et al.. Comparison of increasingly detailed elicitation methods for the assessment of adverse events in pediatric psychopharmacology. J Am Acad Child Adolesc Psychiatry. 2004; 43: 1488–1496.

adverse effects; side effects; adverse drug reaction; symptom assessment; psychopharmacology; clinical trial methodology

© 2012 Lippincott Williams & Wilkins, Inc.