Steinman, Michael A. MD; Boscardin, Christy K. PhD; Aguayo, Leslie CCMEP; Baron, Robert B. MD, MS
Commercial support for accredited continuing medical education (CME) has quintupled over the past decade. By 2006, commercial support for CME totaled $1.2 billion, accounting for 61% of total income for providers of CME.1–3 Over this same period, concern about commercial influence on CME has resulted in substantial efforts to reduce potential bias in accredited educational activities. A handful of institutions have completely forsaken commercial support, but most mainstream organizations, including the Accreditation Council for Continuing Medical Education (ACCME) and the Association of American Medical Colleges, have chosen to preserve commercial support while adopting increasingly more stringent regulations designed to limit the influence of industry funding on the content and objectivity of CME.4
A cornerstone of efforts to limit industry influence in CME is identifying and rejecting proposals for educational activities that pose an unacceptable risk of bias. To accomplish this, many not-for-profit providers of CME, such as academic medical centers and professional organizations, have adopted screening processes to identify activities at high risk of commercial influence. Historically, many of these screening processes have been based on the personal and collective experience of CME professionals. More recently, attempts have been made to develop standardized instruments to predict bias in CME. The Consortium of Continuing Academic Medical Education has developed a standardized rating instrument to prospectively identify activities at high risk of commercial bias. This instrument, which rates activities on the basis of funding sources, responsibility for course logistics, relationships between the course director and industry, and other such characteristics, was found to have good interrater reliability in simulation testing.5 However, to our knowledge there are no published data about the relationship between CME activity characteristics that can be measured prospectively (e.g., sources of funding, conflicts of interest of course chairpersons, or a risk index including these and other factors) and actual or perceived bias in real-world CME activities.6 Thus, few empiric data are available to judge the performance of mechanisms to prospectively identify CME activities at high risk of bias.
To address this question, we studied over 200 CME activities hosted by a major university provider of CME and evaluated the association between characteristics of these activities and ratings of bias made by course attendees after completing the course. First, we evaluated the extent of perceived bias across the range of CME activities. Next, we sought to determine whether a risk index in current use could prospectively identify activities with a higher degree of perceived commercial bias as reported by participants. Finally, we sought to determine whether a variety of characteristics relating to potential commercial influence were associated with perceived bias at those CME activities.
We collected data on 228 directly sponsored live CME activities hosted by the University of California, San Francisco (UCSF) Office of Continuing Medical Education from 2005 to 2007. Over this time, the UCSF Office of CME followed a highly structured series of practices for reviewing proposals for CME activities. Applications for first-time activities underwent initial screening by senior staff to determine major sources of conflicts of interest. Applications for activities which were considered to have unacceptable risk of commercial influence—for example, those with a single industry supporter—were not considered further.
The remaining applications were then submitted for peer review by faculty members of UCSF's CME Governing Board. Each application was also separately evaluated by office staff using an internally developed rating system for assessing an activity's potential for commercial bias, ultimately scored as “lowest,” “intermediate,” or “highest” (see List 1). After full review of each application, peer reviewers designated each activity as rejected, approved, or approved with audit. Courses that were approved with audit, typically those rated as intermediate or highest risk in the prepeer review, required additional measures before and during the activity to mitigate risk. These measures included discussion with the course chair, advance review of presentation materials, and/or close observation of the CME activities.
At the completion of CME activities, participants were asked to evaluate several characteristics of the activity, including an overall rating of the course (using a five-point Likert scale) and a question about whether the activity was free of commercial bias (answered with a yes/no response). Data from individuals' responses were combined to provide summary statistics for each activity.
In 15 of 228 CME activities, participants were not asked about commercial bias in the course evaluation. Thus, our final analytic cohort comprised 213 CME activities. Compared with our final cohort, the 15 activities without information on perceived bias had similar distribution of risk characteristics and overall quality scores but smaller numbers of participants (median number of participants 75 versus 132, P = .02).
Our main outcome variable was the percentage of respondents at an activity who perceived that the activity was free of commercial bias. Higher numbers indicate less bias; for example, a score of 100% means that all respondents indicated the activity was bias-free. Our primary predictor variable was the UCSF Office of CME's risk score for commercial bias (rated as lowest, intermediate, or highest), as described above and in Box 1. In addition, we separately evaluated each event characteristic used to construct this risk index, including factors such as whether the activity chair had ties with industry, whether registration fees were below market value, and others.
To evaluate the association between these predictor variables and learner-perceived bias, we employed binomial regression using the “binreg” command in Stata 10.0 (StataCorp, College Station, Texas) in which we modeled the outcome variable as the number of respondents indicating commercial bias out of the total number of respondents for each activity. This effectively weights each course in proportion to the number of respondents. We incorporated an overdispersion adjustment to account for intraactivity correlation.7 Using these methods, we separately tested the association between each predictor variable and perceived commercial bias at each event (the outcome variable). In addition, we entered all predictor variables into a regression model to assess whether combinations of risk factors might predict the degree of perceived bias. As a sensitivity analysis, we repeated our core analyses using alternate methods to allow for intraactivity correlation. All methods produced generally similar estimates, and we report only our main analyses.
Finally, we hypothesized that the perception of bias in an activity might be reflected not only in a direct question about bias but also in the perceived overall quality of the activity. Thus, we conducted additional analyses in which our outcome variable was the overall quality of the activity, represented as the mean value of participants' ratings on a five-point Likert scale. For these analyses, we used linear regression models weighted by the number of evaluation respondents for each course. Because of the summative nature of the data on quality scores, our models could not precisely account for variance in responses within activities. As a result, these analyses are likely to have increased sensitivity to detect differences in perceived quality among activities with and without certain event characteristics.
This research was exempted from review by the Committee on Human Research at UCSF and by the Research and Development Committee at the San Francisco VA Medical Center.
Characteristics of 213 CME activities studied are shown in Table 1. The median number of participants per course was 132 (interquartile range [IQR] 92-194). One third (73, or 34%) of courses had no commercial support, with the remaining courses split between varying levels of commercial support. One hundred seventy (80%) activities were considered at lowest risk of commercial bias based on the UCSF Office of CME's risk rating system, 36 (17%) were considered intermediate risk, and 7 (3%) were considered highest risk.
The results of course evaluations are shown in Table 2. The mean response rate was 56% (SD 15%). Few respondents perceived commercial bias, with a median of 97% (IQR 95%-99%) of respondents stating that the course they attended was free of commercial bias (Figure 1). The perceived overall quality of CME activities was high, with a mean rating of 4.4 (SD 0.2) on a five-point Likert scale.
The UCSF Office of CME's risk score for commercial influence was not associated with perceived bias, as we found no significant difference in perceived bias between CME activities classified as lowest, intermediate, and highest risk (P = .63). Similarly, level of commercial support was not associated with perceived bias (P = .90). With the exception of the course director being a first-time chair, none of the individual risk factors we assessed were associated with perceived bias (Table 3). Simultaneous inclusion of all of these variables in a single model yielded limited results, with 3% of the total variance in perceived bias between activities explained by these risk factors.
Next, we evaluated several other event characteristics which we did not a priori hypothesize would be associated with perceived bias. Number of registrants was not associated with perceived bias (P = .46). However, the response rate to course evaluations was associated with perceived bias (P < .001), with an adjusted mean “bias-free” rating of 95.8% in activities with below-median response rates versus 96.9% in those with above-median response rates. There was also a statistical trend for activities conducted during later years of the study to be considered less free of bias (adjusted mean “bias-free” ratings of 97.1% for activities in 2005 versus 96.1% for activities in 2006 and 2007, P = .08). Together, number of registrants, response rate, and course year explained 11% of the total variance in perceived bias between activities. When all other predictor variables were added to the model, the total explained variance was 15%.
Finally, we evaluated whether potential risk factors for commercial bias were associated with the overall quality of the activity. As with perceived bias, neither the risk score nor level of commercial support was associated with the overall quality of the activity. Similarly, the only variable significantly associated with overall quality was the presence of a first-time course chair (mean Likert score 4.23 for activities with first-time chair versus 4.40 for activities without first-time chair, P = .01). When all risk factors used in the risk score were included in a single model, 9% of the variance in overall quality score was explained by these risk factors.
In this study of 213 educational programs organized through an academic provider of CME, the vast majority of CME activities were perceived by participants to be free of commercial bias. The extent of commercial support, a variety of other event characteristics, and a summative risk index were not associated with the level of perceived bias or the perceived overall quality of CME activities. Stated otherwise, rates of perceived commercial bias were consistently low regardless of the presence or absence of risk factors for commercial bias.
There is a paucity of other research that has directly evaluated the impact of commercial support on the content of CME.6 However, other studies of bias in CME and on relationships between industry and medicine can shed light on potential explanations for our findings. The most direct interpretation of our results is that CME activities in general are free of commercial bias. Under rules of the ACCME, all accredited providers of CME must abide by the Council's Standards of Commercial Support. These standards require that accredited activities be balanced and that conflicts of interest be disclosed and managed.8,9 However, these regulations cannot prevent all forms of commercial influence, and experts in drug industry promotion have identified widespread opportunities for commercial influence in CME and have documented industry marketing strategies that rely heavily on clinician education to boost drug sales.10–20 Other reports have documented that many physicians believe that industry-supported CME is biased.13,21,22 Finally, many commentators have noted that it is unlikely that industry would expend $1.2 billion per year to support CME if it did not help companies' bottom lines.1–3,11,23,24 Thus, although one interpretation of our findings is that industry support does not lead to bias in CME, other interpretations are possible as well.
One such alternate explanation is that the screening process instituted by the UCSF Office of CME successfully rejected activities with commercial bias that might have been permitted by other CME providers. This screening process is substantially more rigorous than is prescribed in the ACCME's Standards of Commercial Support. For example, although doing so is not required by the standards, the UCSF Office of CME rejects proposals with a single industry supporter or those with no cost to the participant. This screening process may have weeded out CME activities at higher risk of commercial bias, leaving only activities in which commercial bias was more subtle or entirely absent. Furthermore, activities that were allowed to proceed but flagged as having intermediate or higher risk of bias were subject to mitigation procedures before and/or during the activity.
Finally, our findings of low rates of perceived bias may in part be explained by the insensitivity of the simple “yes/no” question used to assess learners' perceptions of bias. Given that commercial influence is often subtle, a single binary question may fail to fully capture the range of learner perceptions of commercial influence.25 Moreover, learner perceptions of bias may not precisely correspond to actual bias in CME activities. It is possible that some suspicious observers may have perceived bias where, in fact, there was none. However, a wealth of literature in the medical and social sciences suggests that physicians (and people in general) often fail to perceive bias, particularly when these biases are subtle.26–28
Prior work investigating mechanisms of bias in CME has highlighted the importance of subtle, often subconscious and unintentional forms of bias that may be difficult for screening processes to identify and for learners to perceive.1,15,29 Course organizers may focus course curricula more on drug or device-based therapies and less on lifestyle-based treatments, or devote special attention to one aspect of disease management for which a supporter's drug (or class of drugs) may be commonly used.30 In addition, the medical literature from which evidence-based recommendations are made is subject to commercial influence, as noted in studies which have shown strong and consistent associations between industry support of clinical trials and the outcomes and interpretation of those trials.31,32 Speakers may also play an important role in introducing subtle biases, for example, through their choice of topics and presentation of course material.
In each of these cases, bias is introduced not through a direct quid pro quo but through subconscious attitudes and feelings of reciprocity that can arise when a speaker or course director has benefited from industry, for example, through funding of an educational or research grant, service on a speakers' bureau, or receipt of personal educational materials, food, or samples for one's clinic. Because attitudes and feelings of reciprocity largely occur on a subconscious level, physicians often fail to recognize the ways in which they might be influenced and overestimate their ability to resist commercial influence.23,26 Of note, in 2007 only 20% of speakers at UCSF CME activities reported conflicts of interest, such as grant funding or speakers' bureau honoraria relevant to their talks; however, this number may be substantially higher elsewhere.
To the extent that influence on the content of CME can often be subtle and occur without the conscious knowledge of course organizers and speakers, bias in CME can be difficult to detect even for the watchful observer.23,33 In addition, in many cases it can be challenging, if not impossible, to determine whether a speaker is emphasizing a certain topic or recommending a certain treatment strategy because of subtle commercial influence on that speaker or because that speaker has a truly independent, scientifically valid opinion that reaches the same conclusion.22 Paradoxically, CME activities in which commercial influence is subtle are likely to be more effective at changing participants' attitudes and behaviors than activities with overt influence, because physicians are more receptive to information that seems objective and may reject information they perceive to be promotional.30
Given the difficulty of disentangling unvarnished educational goals and scientific opinion from subtle forms of commercial bias, there have been increasing calls to fundamentally change the relationship between the health care industry and CME providers. In a seminal article, Brennan et al29 recommended eliminating industry support of individual CME activities at academic medical centers and instead suggested a mechanism in which industry donations would be pooled into a central account that would be used to help fund individual CME activities without industry input. Several universities have adopted this model, with a handful forsaking all industry funding of their CME programs.34 A recent conference commissioned by the Josiah Macy Foundation went a step further, recommending the complete elimination of industry support of accredited CME.16 Not surprisingly, these recommendations are controversial, in part because many CME providers are financially dependent on industry, and because physicians are concerned about increases in course registration fees if industry support were to be substantially reduced or withdrawn.4,13,21,35
Our results could be interpreted as either supporting or contradicting these recommendations. On the one hand, one might conclude that the safeguards offered by the UCSF Office of CME's screening practices resulted in a series of CME programs with little if any bias. On the other hand, our results are consistent with the observation that commercial bias is difficult to detect, both by organizers and recipients of CME, and that the only way to safely guard against industry influence is to eliminate or limit industry involvement in CME. To resolve this and other questions, more research is needed to systematically investigate the presence and impact of bias in CME, including in-depth explorations of course content and learner perceptions of bias. Of note, definitive research will likely require large sample sizes and sophisticated measures, making it unlikely that a substantial body of new research will be available in the near future. Until then, policy makers will need to carefully interpret existing data and apply lessons learned from research on other types of interactions between physicians and industry to help inform their judgments.
Our study has several limitations. As noted earlier, our sample of CME activities were organized by a single institution with a distinct set of criteria for vetting proposed activities. It is thus unclear to what extent our findings would be generalizable to other academic and nonacademic providers of CME with less stringent review criteria and a different set of administrative procedures and institutional culture. In addition, as noted above, the question we used to determine perceptions of bias was limited to a binary yes/no response, preventing us from capturing subtlety in participant responses. Third, our study evaluated only activities which had passed an initial screening process in which proposals deemed to have an unacceptably high risk of commercial influence were rejected (whereas such activities might be approved by other accredited providers of CME). As a result, our study design prevents us from assessing perceived bias in activities believed to carry the highest risk of industry influence. Finally, our study evaluated a moderate but limited number of CME activities. Nonetheless, it is unlikely that a larger sample of activities would have changed our conclusions, because even if the observed differences in perceived bias between lower- versus higher-risk activities were statistically significant, they would not be meaningfully different.
In summary, in this study of 213 CME activities, rates of perceived commercial bias were very low, with no differences in perceived bias between activities with and without potential risk factors for commercial influence. These findings suggest that rigorous review criteria effectively screened out activities with explicit commercial bias. Further research will be needed to evaluate the presence of subtle forms of commercial bias and risk factors to predict these biases in CME.
The authors thank John Boscardin, PhD, and Kathy Fung, MS, for their help with the statistical analyses.
Funding/Support: Funding for this work was provided by VA Health Services Research and Development Service (CDTA 01-013) and by the National Institute on Aging (K23 AG030999) (both for Dr. Steinman).
Other disclosures: None.
Ethical approval: This study was exempted from review by the Committee on Human Research at the University of California, San Francisco and by the Research and Development Committee at the San Francisco VA Medical Center.
Disclaimer: Dr. Steinman had full access to all data for this study and takes full responsibility for the results. Dr. Steinman served as an unpaid expert witness for the plaintiff in United States ex. rel Franklin versus Pfizer, Inc., litigation which alleged that Pfizer and Parke-David illegally marketed gabapentin (Neurontin) for uses not approved by the FDA, including the use of medical education for marketing purposes. Dr. Steinman also helped to found an online archive of drug industry marketing documents from this and other litigation, including soliciting a gift of start-up funds from the lead lawyer in the gabapentin litigation. In addition, Dr. Steinman contributed unpaid efforts to an educational grant funded with money from an Attorney General Settlement Fund created through an out-of-court settlement of this litigation.
1Steinbrook R. Financial support of continuing medical education. JAMA. 2008;299:1060–1062.
3Steinman MA, Baron RB. Is continuing medical education a drug-promotion tool?: YES. Can Fam Physician. 2007;53:1650–1657.
4Relman AS. Industry support of medical education. JAMA. 2008;300:1071–1073.
5Barnes BE, Cole JG, King CT, et al. A risk stratification tool to assess commercial influences on continuing medical education. J Contin Educ Health Prof. 2007;27:234–240.
7McCullaugh P, Nelder JA. Generalized Linear Models. 2nd ed. London, UK: Chapman and Hall; 1989.
12Relman AS. Separating continuing medical education from pharmaceutical marketing. JAMA. 2001;285:2009–2012.
13Mueller PS, Hook CC, Litin SC. Physician preferences and attitudes regarding industry support of CME programs. Am J Med. 2007;120:281–285.
14Smith JL, Cervero RM, Valentine T. Impact of commercial support on continuing pharmacy education. J Contin Educ Health Prof. 2006;26:302–312.
15Association of American Medical Colleges. Industry Funding of Medical Education: Report of an AAMC Task Force. Washington, DC: Association of American Medical Colleges; June 2008.
16Fletcher SW. Chairman's Summary of the Conference: Continuing Education in the Health Professions: Improving Healthcare Through Lifelong Learning. New York, NY: Josiah Macy, Jr. Foundation; 2008.
17Carlat D. Diagnosis: Conflict of interest. New York Times. June 13, 2007.
18Steinman MA, Bero LA, Chren MM, Landefeld CS. Narrative review: The promotion of gabapentin: An analysis of internal industry documents. Ann Intern Med. 2006;145:284–293.
19Fugh-Berman A, Ahari S. Following the script: How drug reps make friends and influence doctors. PLoS Med. 2007;4:e150.
20Bowman MA, Pearle DL. Changes in drug prescribing patterns related to commercial company funding of continuing medical education. J Contin Educ Health Prof. 1988;8:13–20.
21Rutledge P, Crookes D, McKinstry B, Maxwell SR. Do doctors rely on pharmaceutical industry funding to attend conferences and do they perceive that this creates a bias in their drug selection? Results from a questionnaire survey. Pharmacoepidemiol Drug Saf. 2003;12:663–667.
22Cornish JK, Leist JC. What constitutes commercial bias compared with the personal opinion of experts? J Contin Educ Health Prof. 2006;26:161–167.
23Association of American Medical Colleges. The Scientific Basis of Influence and Reciprocity: A Symposium. Washington, DC: Association of American Medical Colleges; June 12, 2007.
24Casebeer LL, Shillman RS. What's ROI got to do with CME? Med Mark Media. 2002:101–105.
25Takhar J, Dixon D, Donahue J, et al. Developing an instrument to measure bias in CME. J Contin Educ Health Prof. 2007;27:118–123.
26Pronin E. Perception and misperception of bias in human judgment. Trends Cogn Sci. 2007;11:37–43.
27Chren MM, Landefeld CS, Murray TH. Doctors, drug companies, and gifts. JAMA. 1989;262:3448–3451.
28Dana J, Loewenstein G. A social science perspective on gifts to physicians from industry. JAMA. 2003;290:252–255.
29Brennan TA, Rothman DJ, Blank L, et al. Health industry practices that create conflicts of interest: A policy proposal for academic medical centers. JAMA. 2006;295:429–433.
30Fugh-Berman A, Batt S. “This may sting a bit”: Cutting CME's ties to pharma. Virtual Mentor. 2006;8:412–415.
31Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ. 2003;326:1167–1170.
32Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: A reflection of treatment effect or adverse events? JAMA. 2003;290:921–928.
33Avorn J, Chen M, Hartley R. Scientific versus commercial sources of influence on the prescribing behavior of physicians. Am J Med. 1982;73:4–8.
34Harris G. Stanford to limit drug maker financing. New York Times. August 26, 2008.