Secondary Logo

Share this article on:

The Relationship Between Physician Participation in Continuing Professional Development Programs and Physician In-Practice Peer Assessments

Wenghofer, Elizabeth F. PhD; Marlow, Bernard MD; Campbell, Craig MD; Carter, Lorraine PhD; Kam, Sophia MA; McCauley, William MD; Hill, Lori MEd

doi: 10.1097/ACM.0000000000000243
Research Reports

Purpose To investigate the relationship between physicians’ performance, as evaluated through in-practice peer assessments, and their participation in continuing professional development (CPD).

Method The authors examined the predictive effects of participating in the CPD programs of the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians of Canada one year before in-practice peer assessments conducted by the medical regulatory authority in Ontario, Canada, in 2008–2009. Two multivariate logistic regression models were used to determine whether physicians who reported participating in any CPD and group-based, assessment-based, and/or self-directed CPD activities were more or less likely to receive satisfactory assessments than physicians who had not. All models were adjusted for the effects of sex, age, specialty certification, practice location, number of patient visits per week, hours worked per week, and international medical graduate status.

Results A total of 617 physicians were included in the study. Analysis revealed that physicians who reported participating in any CPD activities were significantly more likely (odds ratio [OR] = 2.5; P = .021) to have satisfactory assessments than those who had not. In addition, physicians participating in group-based CPD activities were more likely to have satisfactory assessments than those who did not (OR = 2.4; P = .016).

Conclusions There is encouraging evidence supporting a positive predictive association between participating in CPD and performance on in-practice peer assessments. The findings have potential implications for policies which require physicians to participate in programs of lifelong learning.

Dr. Wenghofer is associate professor, School of Rural and Northern Health, and Northern Ontario School of Medicine, Laurentian University, Sudbury, Ontario, Canada.

Dr. Marlow is assistant professor, Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada.

Dr. Campbell is director of professional affairs, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada, and associate professor, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.

Dr. Carter is director, Centre for Flexible Learning, Nipissing University, North Bay, Ontario, Canada, and professor, Northern Ontario School of Medicine, Sudbury, Ontario, Canada.

Ms. Kam is a PhD student, School of Rural and Northern Health, Laurentian University, Sudbury, Ontario, Canada.

Dr. McCauley is medical advisor, Quality Management Division, College of Physicians and Surgeons of Ontario, Toronto, Ontario, Canada, and associate professor, Schulich School of Medicine & Dentistry, University of Western Ontario, London, Ontario, Canada.

Ms. Hill is former manager, Continuing Professional Development, College of Family Physicians of Canada, Mississauga, Ontario, Canada.

Funding/Support: National Association of Faculties of Medicine of Canada (AFMC) Standing Committee on Continuing Medical Education Continuing Medical Education/Continuing Professional Development Research Fund.

Other disclosures: None reported.

Ethical approval: Ethical approval for the study was provided by the Laurentian University research ethics board.

Disclaimer: The AFMC had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The College of Physicians and Surgeons of Ontario, the College of Family Physicians of Canada, and the Royal College of Physicians and Surgeons of Canada, as organizations, were not involved in the approval of this article. Coauthors from the College of Physicians and Surgeons of Ontario (Dr. McCauley), the College of Family Physicians of Canada (Dr. Marlow, formerly, and Ms. Hill), and the Royal College of Physicians and Surgeons of Canada (Dr. Campbell) provided oversight for the retrieval, linkage, and anonymization of data from their respective institutions, as well as input on the design and conduct of the study; collection, management, and interpretation of the data; and preparation, review, and approval of the manuscript.

Previous presentations: A summary of the preliminary findings of this study was presented at the Annual Spring Meeting of the Coalition for Physician Enhancement, June 7–8, 2012, Denver Colorado; and CME Congress, May 31 to June 2, 2012, Toronto, Ontario.

Correspondence should be addressed to Dr. Wenghofer, School of Rural and Northern Health, Laurentian University, Ramsey Lake Road, Sudbury, Ontario, Canada, P3E 2C6; telephone: (705) 675-1151 ext. 3925; e-mail: ewenghofer@laurentian.ca.

Continuing professional development (CPD) encompasses a broad range of lifelong learning, skill development, and/or educational activities aimed at maintaining and improving physicians’ practice, professional behaviors, clinical skills and knowledge, and ultimately, care provided to patients. Physician participation in CPD has long been considered a professional obligation in Canada and the United States to sustain competence, enhance performance, and demonstrate accountability to the public for the privilege of self-regulation.1–10 The value placed on CPD in relation to physician performance is evident in policies of the Canadian specialty certification bodies. The College of Family Physicians of Canada (CFPC) and the Royal College of Physicians and Surgeons of Canada (Royal College) have mandated participation in CPD as a requirement for maintenance of membership and fellowship. In addition, the medical regulatory authorities of many provinces, including the College of Physicians and Surgeons of Ontario (CPSO), have recently made participation in recognized programs of CPD mandatory for all licensed physicians. Such participation is regarded to be a mechanism to ensure an ongoing high level of physician performance for the public.11 On the national level in Canada, the official position of the Federation of Medical Regulatory Authorities of Canada is that participation in CPD is critical to ensuring that the medical profession continues to act in the public’s best interest.12 Specialty certification bodies in the United States rely on CPD for maintenance of physicians’ competence and stress the importance of lifelong learning. The recertification programs of the American Board of Medical Specialties, in part, evaluate maintenance of competence through CPD.10 Similarly, the Federation of State Medical Boards promotes the importance of CPD as part of the assessment of competence for maintenance of licensure.13

The positions of these numerous American and Canadian bodies make clear the importance placed on CPD for sustaining effective performance throughout a physician’s career. At the same time, some scholars question the effectiveness of the CPD approach for ensuring continuing competence of physicians in practice.14 Studies have demonstrated that certain methods of CPD are more effective (e.g., practice-based small-group learning) than others in generating a positive impact on physician performance and care.15–21 Researchers have found, though, that the CPD approaches shown to have a lesser effect on practice behavior (e.g., didactic large-group sessions) tend to dominate educational offerings and, as such, are selected more frequently by physicians than other forms of learning.22–25 Throughout the literature, there is a call for further research on the influence of various types of CPD on practice. This need is increasingly urgent given the current emphasis on lifelong learning as a requirement for maintenance of competence by certification and licensing bodies.1,6,9,26–32

Our study is a contribution to address this gap in the research investigating the overall effect of participation in CPD on physician performance. Our study uses CPD participation data provided by the CFPC and the Royal College and performance data provided by in-practice peer assessments conducted by the CPSO. In analyzing these data, we explored the relationship between physicians’ performances, as evaluated through in-practice peer assessments, and the physicians’ participation in CPD overall, and different types of CPD.

The conceptual framework we used in this study defines physician performance as contextual, insofar as performance reflects not only the physician’s education, credentials, and participation in CPD but also his or her ability to meet the unique requirements of the broader practice environment.33 Within this framework, the performance requirements for one practice setting may vary from those in another, because the scope of practice may differ between locations.

Previous research has indicated that physician clinical performance is multidimensional and can best be conceptualized and measured in terms of different types of physician–patient encounters.34,35 In addition, previous studies have demonstrated that physician demographics, organizational features, and system factors affect physician performance.33,36–38 In our study, we recognize that physician performance is influenced by the types of physician–patient encounters within a practice (i.e., who they see), physicians’ personal characteristics (i.e., their demographics and education), and their practice environments (i.e., where they practice and the resources available to them). Taken together, these components define a physician’s scope of practice.

The practices of two family physicians are described to illustrate this point. Physician 1 is engaged in an urban urgent care/walk-in office practice and also does some long-term care facility duties. He does not have hospital privileges and does not do minor in-office surgical procedures or any surgical assisting. Physician 2 has a full office practice in a small isolated community with very few professional supports (i.e., no other physicians, nurse practitioners, or dietitians). Her practice includes minor in-office surgical procedures and obstetrical work, and she has full hospital privileges and works in the hospital’s outpatient clinic. Although both physician 1 and physician 2 are certified family physicians, it is obvious that their scope of practice is not the same because both their patient needs and practice environments are distinctly different, resulting in different competency and performance requirements.

We considered this conceptual model when structuring our analyses and in the selection of our covariates described in the statistical models below.

Back to Top | Article Outline

Background

In Canada, the CFPC and Royal College are the certifying bodies for, respectively, family physicians and all other specialties. Once certified, physicians, in order to maintain their membership and fellowship, must comply with the CPD requirements of these organizations. The Royal College’s maintenance of certification (MOC) program is a mandatory CPD system for all specialists through which they maintain membership and use of the designation Fellow of the Royal College of Physicians or Fellow of the Royal College of Surgeons. The purpose of the Royal College MOC program is to encourage, support, and enable a learning culture based on reflection, inquiry, peer review, and assessment of knowledge, competence, and performance across the entire spectrum of professional practice roles and competencies outlined in the CanMEDs framework. Each specialist is required to develop a needs-based, practice-specific CPD plan; to document learning activities; and to identify outcomes that have affected their practice. All specialists are required to complete at least 40 credits per year and 400 credits over each five-year cycle.39

Although the CFPC also has a CPD requirement through its MAINPRO (maintenance of proficiency) program and is founded on concepts similar to those in the Royal College MOC program, there are differences in how the programs are structured. The MAINPRO program began in 1995 and is undergoing major revisions. There are currently three categories of credit: M1 activities, which are primarily group and self-learning activities that have been peer reviewed and accredited; M2 activities, which are primarily composed of unaccredited group and self-learning activities; and MAINPRO-C activities, which are based on individual needs and include reflective activities or exercises. As of January 1, 2013, members must complete at least 25 credits per year and a minimum of 250 credits every five years to maintain membership and/or certification (earning the CFPC designation). Fellows of the CFPC (designated FCFP) must also complete a minimum of 25 MAINPRO-C credits in every five-year cycle to maintain this special designation.40

In Ontario, the CPSO’s peer assessment program started in 1980. Any physician less than 70 years of age who has been practicing for 5 years or more is included in the random selection pool. Physicians in their 70th year of age are automatically assessed and then assessed again every 5 years thereafter, for as long as they stay in active practice.

Back to Top | Article Outline

Method

Peer assessment process and physician performance data

All physicians licensed for independent practice with primary practice addresses in Ontario, Canada, and who were randomly selected to participate in a peer assessment by the CPSO in 2008 and 2009, constituted the study sample.

Each CPSO peer assessment is performed by a single, trained, practicing physician who has been previously assessed and achieved an excellent rating. The assessment consists of an in-practice review of 20 to 30 medical records, guided by a standardized series of protocol forms, which enables the assessor to develop a picture of the physician’s practice and understand the physician’s approach to patient care. The assessor collects information regarding various dimensions of care, including management of patients with acute conditions, patients with chronic conditions, preventive care, and psychosocial care. The assessor also reviews care activities such as taking adequate histories, conducting appropriate examinations, ordering necessary diagnostic tests, identifying the appropriate course of action, conducting necessary interventions, and monitoring patients. The record review is followed by a discussion with the physician. The assessor then completes the protocol forms, using the information from the records and the discussion. In addition to making recommendations for improvement, the assessor makes note of areas in which the physician demonstrates exemplary care (Figure 1).

Figure 1

Figure 1

The adjudication of the report is conducted by the Quality Assurance Committee of the CPSO, which reviews the information provided by the assessor and then assigns one of three outcomes: satisfactory, requiring no further action; reassessment required, to follow up on minor concerns; and serious concerns, which require the physician to attend an interview with the committee to establish a remediation plan. For the purposes of this study, we dichotomized assessment outcomes into satisfactory (no further action) and unsatisfactory (requiring reassessment or interview). Additional details regarding the CPSO peer assessment can be found at the CPSO Web site41 and throughout the literature.33,34,42–46

Back to Top | Article Outline

CPD data

CPD data included all CPD activities reported to the CFPC and Royal College by the assessment physician cohort in the year before their assessment (i.e., 2007 or 2008). Table 1 presents an overview of the various CPD activities for which physicians receive credit from the CFPC and Royal College toward fulfillment of their CPD requirements. We categorized the CPD activities into three broad CPD types: group-based activities, self-directed learning, and assessment-based activities. The categorization activity was completed collaboratively with CFPC and Royal College program experts so that similar and comparable activities could be identified and grouped appropriately. These categories enabled us to compare CPD groupings across the Royal College and CFPC program data.

Table 1

Table 1

We focused our analysis of CPD on reported participation, noted as a dichotomous variable (yes/no). Individuals who reported participating in any CPD activity in the year before assessment were coded as “yes” for having participated in CPD overall. Additionally, physicians who reported participating in any group-based, self-directed learning, or assessment-based CPD activities were coded as “yes” for each of the CPD categories that applied.

Back to Top | Article Outline

Covariates

To be consistent with our conceptual, and in order to remove potential confounders that may influence the relationship between CPD and assessment performance, we targeted covariates that have been identified in the literature as affecting physician performance. More specifically, we included age,33,42–44,47–52 sex,33,43,45,53–55 international medical graduate (IMG) status,37,44,54,55 certification,33,43,51,52,54,55 hours worked and number of patients seen in a typical week,33,56–58 and practice location for those physicians working in Northern Ontario.33,59,60 These data were derived from the CPSO register and annual membership renewal survey, which provides personal information and practice environment details.

Back to Top | Article Outline

Data linkages and ethical approval

Third-party data linkage procedures were used to link each physician’s overall performance data from the CPSO peer assessment program with the CPD data submitted to the CFPC or Royal College (i.e., data sets were linked by researchers at Laurentian University). All personal identifiers were removed from the data, and encrypted study identification numbers were assigned to study participants. Ethical approval for the study was provided by the Laurentian University research ethics board in Sudbury, Ontario.

Back to Top | Article Outline

Analysis

We calculated frequency distributions of all variables to provide an overview of the study sample and to examine the data for quality, completeness, and potential outliers. Chi-square analysis was used to determine the univariate relationship between reported participation in CPD and types of CPD and peer assessment outcome. We used multivariate logistic regression to determine whether reported participation in CPD was predictive of those physicians assigned a “satisfactory” assessment in contrast with those assigned an “unsatisfactory” outcome. Two logistic regression models were completed. Model 1 examined the effects of overall reported participation in CPD (yes/no) on satisfactory performance. Model 2 examined the effects on satisfactory performance of reported participation in specific categories of CPD, including group-based activities (yes/no), self-directed learning (yes/no), and assessment-based activities (yes/no). Both models considered for inclusion all covariates outlined above. In addition to the listed covariates, we included an interaction term between reported CPD participation and certification (Royal College versus CFPC) in the variable selection process. Although the CFPC’s and Royal College’s mandatory CPD programs have many comparable features, there may be nuanced differences, which an interaction term could reveal.

We used a backward stepwise conditional entry for both models where P ≤ .05 was required for variable entry into the model, and P ≥ .10 was the threshold for removal. Odds ratios (ORs) and 95% confidence intervals were calculated for all significant predictors.

Back to Top | Article Outline

Results

The final sample size included 617 physicians. Of this group, 580 (94.0%) received a satisfactory peer assessment. Of the 617 physicians, 243 (39.4%) were family physicians, 219 (35.5%) were female, 439 (71.2%) were under the age of 55, and 101 (16.4%) were IMGs. Regarding hours worked and patient visits per week, 280 (47.0%) reported working more than 40 hours a week, and 360 (60.4%) saw fewer than 100 patients per week (Table 2).

Table 2

Table 2

Participation in CPD in the year before their assessment was reported by 465 (77.5%) physicians; 441 (73.5%) reported group-based CPD, and 398 (66.3%) reported participation in more than one type of CPD activity. Although self-directed and assessment-based activities were reported, the numbers of physicians who participated in these categories were considerably lower at 292 (48.7%) and 108 (18.0%), respectively (Table 3). Chi-square analyses indicated significantly lower proportions of unsatisfactory assessments in physicians who participated in CPD overall (χ2 = 5.32; P = .021), participated in a variety of CPD activities as opposed to no reported CPD or only one type of CPD reported (χ2 = 9.414; P = .009), and participated in group-based CPD (χ2 = 7.655; P = .006).

Table 3

Table 3

We used logistic regression to analyze the data according to the two models previously described. The regression for model 1 revealed that physicians who reported participating in CPD activities in the year before assessment were more likely (OR = 2.5; P = .021) to have a satisfactory assessment than those physicians who did not report participation in CPD (Table 4).

Table 4

Table 4

The regression for model 2 indicated that physicians who reported participation in group-based CPD were more likely to have satisfactory assessments than those who did not report participation in group-based activities (OR = 2.4, P = .016) (Table 4). Reported participation in self-directed learning activities, assessment-based activities, and the interaction terms were not significant predictors of satisfactory performance in either model.

In both models, males were significantly less likely to do well on assessment than females, and Royal College–certified specialists were more likely to do well on assessment than CFPC-certified family practitioners. Both findings are consistent with previous studies on the CPSO peer assessment program.33,42–44,46

Back to Top | Article Outline

Discussion

In this study, we examined the predictive relationship between reported participation in CPD, different types of CPD, and practice performance as determined through in-practice peer assessment. We found a positive predictive effect of reported participation in CPD on physician performance for assessments completed in the year following the reported CPD activities. In addition, we found that group-based activities had a positive predictive effect on performance, whereas self-directed learning and assessment-based activities were not predictive of performance as determined by in-practice peer assessments.

Our findings suggest a positive relationship between CPD participation and physician performance in practice, providing further evidence for the policies of medical regulators, certifying bodies, and CPD provider organizations which require or promote participation in CPD activities. However, there are some caveats to consider when interpreting our findings.

First, the CPD data are based on self-reported activities. As mentioned earlier, physicians report their CPD activities to the CFPC and Royal College over a five-year cycle. Some physicians will retroactively report participation in learning activities throughout the five years and not necessarily during the year the activity was completed. Therefore, the data likely represent only those CPD activities that physicians have completed and reported; they may not capture all completed learning activities. For this reason, we chose to examine the participation in any CPD rather than to focus on amounts of CPD. Future studies should focus on total CPD activities (e.g., total credits) reported across a complete cycle and examine whether or not the timing of reporting in the cycle (e.g., reporting of all activities in the weeks before the end of the cycle versus regular reporting of activities throughout the cycle) is associated with satisfactory performance. The relationship between CPD and performance also needs to be examined for a threshold effect, to determine whether there is a minimum level of CPD required to support satisfactory performance.

Second, our research team worked cooperatively to construct the categories of CPD types used in this study. Although we concluded that these broad categories were appropriate categorizations of the range of accredited learning activities, the question may be raised as to whether our allocation of individual CPD activities to each category could have influenced the results. However, our choice of allocation for a specific CPD activity would not likely have affected our overall findings because our focus was on whether reported participation in any CPD activity affected performance. In future work, we hope to recruit a sufficient sample size to examine in greater detail the impact of individual types of CPD activities (e.g., practice-based small-group learning) on practice performance.

Our third caveat, and related to the point above, is that our findings indicate a significant positive predictive relationship with group-based activities but not with self-directed or assessment-based activities. This finding was somewhat unexpected and in contradiction to the literature. The lack of a significant predictive effect of self-directed and assessment-based CPD may be due to these activities’ representing a far smaller proportion of the total CPD activities compared with group-based activities. It would be well worth revisiting our analysis with CPD program data reported from a full five-year cycle, which could provide a more robust sample of self-directed and assessment-based activities.

Further, although we focused on covariates that have been shown to influence physician practice performance, our list is not exhaustive. In future studies, other important performance factors (e.g., remuneration) should be explored because these may significantly influence the results presented here.

Finally, some readers might note the very high percentage (94%) of physicians who did well on assessment and question the representativeness of this sample. Previous publications based on the CPSO peer assessment program have shown that randomly selected family physicians typically do exceptionally well on assessment, with approximately 85% achieving satisfactory assessments.61 Given those findings, a rate of 94% for satisfactory assessments might appear high. However, this finding might be partially explained by the fact that Royal College–certified physicians tend to perform better on assessment than CFPC-certified physicians,61 and 60% of the study’s sample was composed of specialists certified by the Royal College. Nonetheless, even in this high-performing population, we still find a considerable effect, an increase in the probability of a positive outcome with CPD participation, lending further confidence to our findings.

Notwithstanding these caveats, our findings contribute to the recently growing body of evidence62 supporting the premise that physicians who participate in CPD demonstrate satisfactory competence and performance in their practice. However, our data do not allow us to draw causal links between CPD participation and performance. As well, this study cannot allow us to conclude that the actual CPD activities provided target the skills, knowledge, and attitudes that physicians require to maintain ongoing levels of competence and performance in practice or whether there are other factors contributing to our results. For example, perhaps physicians who regularly participate in CPD activities hold particular commitments to lifelong learning or attitudes toward medical practice that are the actual causal factors driving good performance. Similarly, it may be that physicians who participate in CPD activities structure their practices in particular ways that support competent performance in practice. Future studies are needed to investigate these and other possibilities.

In the next phases of our research, our team will examine the predictive effects of engaging in specific types of CPD activities and performance within specific dimensions of care (e.g., chronic condition management, preventive care and health maintenance, acute condition management). We will also explore the relationships between participation in CPD activities and concerns from the public, including overall and specific complaints (e.g., communication, professionalism, quality of care). We hope this work will further add to understanding the impact of CPD on important dimensions of clinical care.

In the meantime, individual physicians who participate in lifelong learning should be encouraged to know that their participation in CPD is a positive predictor of performance in practice reviews. CPD planners and providers can likewise be assured that their work holds potential for ensuring the satisfactory performance of physicians. Lastly, our study provides important evidence to support the current policy directions of various Canadian and American certification and regulatory bodies that mandate CPD and lifelong learning. In an age where the public demands increasing transparency and accountability and for extending the responsibility of self-regulation to the medical profession, all evidence, in support or in opposition of mandatory CPD, is important.

Back to Top | Article Outline

References

1. Lowe MM, Aparicio A, Galbraith R, Dorman T, Dellert EAmerican College of Chest Physicians Health and Science Policy Committee. . The future of continuing medical education: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135(3 suppl):69S–75S
2. Frank JR The CanMEDS 2005 Physician Competency Framework. 2005 Ottawa, Ontario, Canada Royal College of Physicians and Surgeons of Canada
3. Canadian Medical Association. CMA Policy Statement: CMA Code of Ethics. 2004 Ottawa, Ontario, Canada Canadian Medical Association http://policybase.cma.ca/dbtw-wpd/PolicyPDF/PD04-06.pdf. Accessed February 15, 2014
4. du Boulay C. From CME to CPD: Getting better at getting better? BMJ. 2000;320:393–394
5. Shaw K, Cassel CK, Black C, Levinson W. Shared medical regulation in a time of increasing calls for accountability and transparency: Comparison of recertification in the United States, Canada, and the United Kingdom. JAMA. 2009;302:2008–2014
6. Chaudhry HJ, Talmage LA, Alguire PC, Cain FE, Waters S, Rhyne JA. Maintenance of licensure: Supporting a physician’s commitment to lifelong learning. Ann Intern Med. 2012;157:287–289
7. Chaudhry HJ, Rhyne J, Cain FE, Young A, Crane M, Bush F. Maintenance of licensure: Protecting the public, promoting quality health care. J Med Regul. 2010;96:13–20 http://www.fsmb.org/pdf/mol-bg.pdf. Accessed February 15, 2014
8. Levinson W, Holmboe E. Maintenance of certification: 20 years later. Am J Med. 2011;124:180–185
9. Miller SH, Thompson JN, Mazmanian PE, et al. Continuing medical education, professional development, and requirements for medical licensure: A white paper of the Conjoint Committee on Continuing Medical Education. J Contin Educ Health Prof. 2008;28:95–98
10. American Board of Medical Specialties. . ABMS maintenance of certification. http://www.abms.org/Maintenance_of_Certification/ABMS_MOC.aspx. Accessed February 15, 2014
11. College of Physicians and Surgeons of Ontario. . Consultation on proposed amendments to the QA regulations. Dialogue. 2009 http://www.cpso.on.ca/uploadedfiles/policies/consultations/QAamendments_Sept09.pdf?terms=CPD+require. Accessed February 15, 2014
12. Federation of Medical Regulatory Authorities of Canada Revalidation Working Group. Physician Revalidation: Maintaining Competence and Performance [position paper]. 2007 Federation of Medical Regulatory Authorities of Canada 4 http://www.fmrac.ca/committees/documents/final_reval_position_eng.pdf. Accessed February 15, 2014
13. Federation of State Medical Boards. Report of the Board of Directors: Maintenance of Licensure (BRD RPT 10-3). http://www.fsmb.org/pdf/BD_RPT_10-3_MOL.pdf. Accessed February 15, 2014
14. Levinson W. Revalidation of physicians in Canada: Are we passing the test? CMAJ. 2008;179:979–980
15. Norman GR, Shannon SI, Marrin ML. The need for needs assessment in continuing medical education. BMJ. 2004;328:999–1001
16. Borgiel AE, Williams JI, Davis DA, et al. Evaluating the effectiveness of 2 educational interventions in family practice. CMAJ. 1999;161:965–970
17. Grol R. Improving the quality of medical care: Building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001;286:2578–2585
18. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995;274:700–705
19. Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: A review of systematic reviews. Int J Technol Assess Health Care. 2005;21:380–385
20. Davis D, Galbraith RAmerican College of Chest Physicians Health and Science Policy Committee. . Continuing medical education effect on practice performance: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135(3 suppl):42S–48S
21. Scott IA, Phelps G, Brand C. Assessing individual clinical performance: A primer for physicians. Intern Med J. 2011;41:144–155
22. Kühne-Eversmann L, Eversmann T, Fischer MR. Team- and case-based learning to activate participants and enhance knowledge: An evaluation of seminars in Germany. J Contin Educ Health Prof. 2008;28:165–171
23. Gray J. Changing physician prescribing behaviour. Can J Clin Pharmacol. 2006;13:e81–e84
24. Duffy FD, Holmboe ES. Self-assessment in lifelong learning and improving performance in practice: Physician know thyself. JAMA. 2006;296:1137–1139
25. Byrick RJ, Naik VN, Wynands JE. Simulation-based education in Canada: Will anesthesia lead in the future? Can J Anaesth. 2009;56:273–275
26. American Board of Medical Specialties. . The value of ABMS MOC. http://www.abms.org/Maintenance_of_Certification/value_of_MOC.aspx. Accessed February 15, 2014
27. Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit and feedback: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006(2):000259
28. O’Brien MA, Rogers S, Jamtvedt G, et al. Educational outreach visits: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007(4):CD000409
29. Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007;27:6–15
30. Bennett NL, Davis DA, Easterling WE Jr, et al. Continuing medical education: A new vision of the professional development of physicians. Acad Med. 2000;75:1167–1172
31. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287:226–235
32. Naik VN, Wong AK, Hamstra SJ. Review article: Leading the future: Guiding two predominant paradigm shifts in medical education through scholarship. Can J Anaesth. 2012;59:213–223
33. Wenghofer EF, Williams AP, Klass DJ. Factors affecting physician performance: Implications for performance improvement and governance. Healthc Policy. 2009;5:e141–e160
34. Wenghofer EF, Williams AP, Klass DJ, Faulkner D. Physician–patient encounters: The structure of performance in family and general office practice. J Contin Educ Health Prof. 2006;26:285–293
35. Klass D. A performance-based conception of competence is changing the regulation of physicians’ professional behavior. Acad Med. 2007;82:529–535
36. Blasier RB. The problem of the aging surgeon: When surgeon age becomes a surgical risk factor. Clin Orthop Relat Res. 2009;467:402–411
37. Lipner R, Song H, Biester T, Rhodes R. Factors that influence general internists’ and surgeons’ performance on maintenance of certification exams. Acad Med. 2011;86:53–58
38. Dahrouge S, Hogg WE, Russell G, et al. Impact of remuneration and organizational factors on completing preventive manoeuvres in primary care practices. CMAJ. 2012;184:E135–E143
39. Royal College of Physicians and Surgeons of Canada. . Maintenance of certification. www.royalcollege.ca/portal/page/portal/rc/members/moc. Accessed February 15, 2014
40. College of Family Physicians of Canada. . Introduction to MAINPRO (maintenance of proficiency). http://www.cfpc.ca/MAINPRO/. Accessed February 15, 2014
41. College of Physicians and Surgeons of Ontario. www.cpso.on.ca. Accessed February 22, 2014
42. Norton PG, Faulkner D. A longitudinal study of performance of physicians’ office practices: Data from the peer assessment program in Ontario, Canada. Jt Comm J Qual Improv. 1999;25:252–258
43. Norton PG, Dunn EV, Soberman L. What factors affect quality of care? Using the peer assessment program in Ontario family practices. Can Fam Physician. 1997;43:1739–1744
44. Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med Educ.. 2009;43:1166–1173
45. Norton PG, Dunn EV, Soberman L. Family practice in Ontario. How physician demographics affect practice patterns. Can Fam Physician. 1994;40:249–256
46. McAuley RG, Paul WM, Morrison GH, Beckett RF, Goldsmith CH. Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario. CMAJ. 1990;143:1193–1199
47. Grace ES, Korinek EJ, Weitzel LB, Wentz DK. Physicians reentering clinical practice: Characteristics and clinical abilities. J Contin Educ Health Prof. 2011;31:49–55
48. Grace ES, Korinek EJ, Tran ZV. Characteristics of physicians referred for a competence assessment: A comparison of state medical board and hospital referred physicians. J Med Regul. 2011;96:8–15
49. Turnbull J, Carbotte R, Hanna E, et al. Cognitive difficulty in physicians. Acad Med. 2000;75:177–181
50. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273
51. Caulford PG, Lamb SB, Kaigas TB, Hanna E, Norman GR, Davis DA. Physician incompetence: Specific problems and predictors. Acad Med. 1994;69(10 suppl):S16–S18
52. Norman GR, Davis DA, Lamb S, Hanna E, Caulford P, Kaigas T. Competency assessment of primary care physicians as part of a peer review program. JAMA. 1993;270:1046–1051
53. Slade S, Busing N. Weekly work hours and clinical activities of Canadian family physicians: Results of the 1997/98 National Family Physician Survey of the College of Family Physicians of Canada. CMAJ. 2002;166:1407–1411
54. Reid RO, Friedberg MW, Adams JL, McGlynn EA, Mehrotra A. Associations between physician characteristics and quality of care. Arch Intern Med. 2010;170:1442–1449
55. Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007;298:993–1001
56. Goedhuys J, Rethans JJ. On the relationship between the efficiency and the quality of the consultation. A validity study. Fam Pract.. 2001;18:592–596
57. Freeman GK, Horder JP, Howie JG, et al. Evolving general practice consultation in Britain: Issues of length and context. BMJ. 2002;324:880–882
58. Howie JG, Heaney DJ, Maxwell M. Measuring quality in general practice: Pilot study of a needs, process and outcome measure. Occas Pap R Coll Gen Pract. 1997;75:1–32, i–xii
59. Chan BTB, Schultz SE Supply and Utilization of General Practitioner and Family Physician Services in Ontario: ICES Investigative Report. 2005 Toronto, Ontario, Canada Institute for Clinical Evaluative Sciences
60. Tepper J, Schultz SE, Rothwell DM, Chan BTB Physician Services in Rural and Northern Ontario: ICES Investigative Report. 2006 Toronto, Ontario, Canada Institute for Clinical Evaluative Sciences
61. College of Physicians and Surgeons of Ontario. Annual Report 2011. Toronto, Ontario, Canada College of Physicians and Surgeons of Ontario http://www.cpso.on.ca/uploadedFiles/policies/publications/AR11.pdf. Accessed February 15, 2014
62. Goulet F, Hudon E, Gagnon R, Gauvin E, Lemire F, Arsenault I. Effects of continuing professional development on clinical performance: Results of a study involving family practitioners in Quebec. Can Fam Physician. 2013;59:518–525
© 2014 by the Association of American Medical Colleges