Brown Medical School in 1996 inaugurated a competency-based curriculum in which graduates in the class of 2000 and beyond were required to demonstrate competence in each of nine abilities (see List 1). A detailed description of the processes used in planning and implementing the new curriculum is available elsewhere.1 In this article we describe how faculty assess students' performances relative to the nine abilities.
Each ability includes a list of criteria that the student is expected to meet at a certain level of achievement, depending upon the student's stage of professional development. The three levels are beginning, intermediate, and advanced.
The beginning level represents the level of ability that should be achieved by a student prior to entering the clinical portion of the medical curriculum. For example, students meet the beginning level in Ability III, Using Basic Science in the Practice of Medicine, through preclinical course work that helps them recognize or identify health problems. Students are expected to achieve the intermediate level by the end of their third-year clerkships. To gain intermediate Ability III competency, students are expected to evince a thorough understanding of a health problem in the clinical setting. The advanced level is achieved through the subinternship, a longitudinal ambulatory clerkship, and fourth-year electives. To meet the criteria for Ability III at the advanced level, a student needs to demonstrate understanding and anticipation of the long-term natural and managed course of a chronic health problem.
Students have opportunities to complete some of the beginning-level competencies as undergraduates. Brown Medical School has an eight-year continuum in which students are accepted as first-year undergraduate students into the Program in Liberal Medical Education (PLME).
Students must attain both beginning and intermediate levels of proficiency in all nine abilities before graduation. In addition, students are required to attain an advanced level of proficiency in Ability IX, Problem Solving, and three other abilities based upon their own interests.
The number of certifications needed by students to demonstrate competence at each level for each ability is determined by the Medical Curriculum Committee. A computerized management program keeps track of students' progress and can be accessed by students over the Internet. Table 1 provides an overview of the many learning experiences that are available for students to earn competency credit at each level for one of the abilities, Ability VII, The Social and Community Contexts of Health Care.
Assessment committees composed of students, faculty, and administrators are empowered by the Medical Curriculum Committee to grant certifying authority to individual courses for specific abilities at designated levels of proficiency. Course leaders request certifying authority by petition to the respective assessment committees. The assessment committees periodically review the courses to assure that courses continue to use appropriate methods of assessment for the particular abilities being assessed.
Following best practice in the field of evaluation, our faculty employ multiple measures in assessing students' competence. Like other competency-based curriculum initiatives such as the new Accreditation Council for Graduate Medical Education (ACGME) Outcome Project with its six general competencies, we emphasize performance-based assessment. Performance-based assessment focuses on capturing and assessing students' actual or demonstrated performances on tasks that are identical or similar to what physicians carry out in their daily practices.
Traditional measures, such as written examinations, remain valuable, particularly in ascertaining students' content mastery. Many of our clerkship directors use combinations of standardized tests such as subject or shelf examinations and performance measures such as objective structured clinical examinations (OSCEs). The shelf examination helps faculty to determine students' mastery of the knowledge base, while the OSCE allows faculty to evaluate students' levels of clinical competence in several intermediate-level abilities. Faculty award course grades and competency certification independently of one another. That is, a student may pass a course based on content knowledge but might fail to earn competency certification, and vice versa.
ASSESSING THE NINE ABILITIES
Ability I—Effective Communication
Assessing students' abilities to communicate through written materials is fairly straightforward. Competency in oral communication is assessed by observing the student, either live or by a recording method such as audiotape or videotape. The observer is either a teacher, a standardized patient, or a real patient trained to assess oral communication. In some situations, the assessor may be an audience that evaluates the effectiveness of a presentation made by a student to a community group. Assessment of communication skills in a group can occur in traditional classroom sessions, in small-group tutorials, in work rounds in hospitals, or as part of teams established for planning or executing projects.
As is the case in most medical schools, communication skills are the focus of assessment in the medical interviewing course. The Brown Interview Checklist is a reliable and valid instrument used to assess students' skills in this course.2 A student's performance is assessed both by direct observation and by an audiotaped interview with a real patient. Communication skills are also assessed in small-group sessions in general pathology and pathophysiology using less detailed rating scales.
The human morphology (anatomy) course assesses communication skills by requiring students to present ten-minute oral “prosections/demonstrations” twice during the course. In this exercise, students relate the dissection to a clinical situation. Course faculty along with visiting clinicians listen to the presentation and then engage the presenting student in a five-minute question-and-answer dialogue. The anatomy faculty rate the students on their communication skills, along with other abilities described below.3
The peer assessment of communication skills is used in an undergraduate elective course on health policy to augment faculty assessment. The teaching assistant gives each student the names of three other students in the class at the beginning of the semester. Each student is then asked to pay particular attention to these three students' class participation according to guidelines (see List 2). At midsemester, each student prepares a short anonymous peer evaluation for the three students. The peer evaluations are collected by the teaching assistant, collated, and given to the students so that each student receives three anonymous peer evaluations. These are never seen by the professor, nor do they figure in the students' grades. These midsemester evaluations appear to be quite effective means of modifying behavior. For example, one student frequently interrupted other students during classroom discussions and tended to dominate discussions. Her behavior changed dramatically for the better after midsemester.
Communication skills are assessed at the intermediate level in each of the clerkships using a common clerkship evaluation form developed by the clerkship directors. OSCEs also assess communication skills relevant to the particular clerkship speciality. In the obstetrics and gynecology clerkship, for example, every student must counsel a woman about contraception, while in the pediatrics clerkship students must demonstrate the ability to communicate with an adolescent. In addition to the individual clerkship OSCEs, students also take a fourth-year OSCE in which communication skills are assessed by the standardized patients.
Ability II—Basic Clinical Skills
Beginning-level competency in this ability is assessed in the second year introduction to clinical medicine course by observing students performing history taking, physical examinations, and basic procedures on real or standardized patients. Real patients with actual physical findings are superior to standardized patients for authenticity, though with proper training standardized patients can do an excellent job of simulating certain physical findings (e.g., many neurologic signs). Standardized patients may be preferred when feedback to the students is desired and in sensitive situations, such as taking a sexual history or performing a genitourinary exam, when it is important that every student be presented with the same challenges.
While checklists are still the predominant method of assessment, recent studies indicate that global rating scales may have greater validity.4 However, data are sparse on the ability of standardized patients to use global rating scales rather than checklists to assess students' history-taking and physical examination skills. Our own experience indicates that, with training, standardized patients can do this well. Standardized patients review videotapes of students doing an OSCE station. Each standardized patient rates the videotape, then discusses with other standardized patients and standardized patient trainers his or her reasons for the decision in as much detail as possible. Using this method, the interrater reliability is quite high.
Procedural skills may also be assessed using real subjects or mannequins. Students first learn to perform procedures on mannequins during a two-day orientation at the beginning of their third year. The students have the procedures described to them, step by step. Then they observe a procedure being done by an expert before doing it themselves.
Certification for Ability II intermediate level in the clerkships requires that students demonstrate proficiency in ten basic procedures and observe and be able to describe ten additional advanced procedures. During the clerkships, students perform procedures under direct and close supervision by someone proficient in the procedure. The supervisor then signs the students' procedure log certifying that the procedure was done in a satisfactory manner. For more advanced diagnostic procedures, such as diagnostic imaging procedures and EKGs, students are not expected to be able to do the procedure, but are expected to interpret the results. In these circumstances, students are presented with an x-ray, EKG, or similar product and asked to interpret it. Clerkship OSCEs and the fourth-year OSCE include such tasks.
In evaluating competency in history-taking performance in the fourth-year OSCE stations that assess this skill, the overall average rating is used to determine passing or failing. The pass/fail cutoff is derived using a compromise standard-setting technique described by Hofstee5 in which faculty estimates of maximally acceptable rates of passing and failing are combined with empirically derived data.
Ability III—Using Basic Science in the Practice of Medicine
Assessment of this ability focuses on how well students can use basic science knowledge to explain clinical phenomena or solve clinical problems. Problem-based learning (PBL) is the primary means of assessing Ability III. In preclinical courses such as pathophysiology, pathology, and medical biochemistry, students use basic science to explain the clinical data provided in cases and to postulate possible mechanisms. In the pathophysiology PBL sessions the cases are presented “cold” to the students at the time of the first small-group meeting. Prior to the next meeting, students list learning issues to be researched. At the next meeting each student presents what he or she has learned. The faculty facilitator assess the students' abilities to use basic science based on these presentations.
Examinations are also used to assess Ability III at the beginning level. Multiple-choice questions (MCQs) using clinical vignettes are an improvement upon traditional MCQs as a means of assessing the degrees to which students can apply to clinical situations the basic science they have learned. Uncued responses, such as short-answer questions, are generally preferred by our basic science faculty despite the additional time required to grade such exams. For example, the anatomy and pathology course leaders believe that traditional long essay and shorter modified essay questions better assess a student's ability to use basic science to explain clinical phenomena.
Take-home examinations have been used in pathophysiology. These exams rely less on memory and more authentically reflect what physicians actually do, that is, “go back to the books” when confronted with complex, atypical, or challenging clinical problems. For example, in the nutrition pathophysiology section students were required to use a computer program called “Food Processor” to calculate the nutritional constituents of a dietary log from a patient, to analyze those results, and to describe the kind of nutritional advice they would give the patient based on the analysis.
The anatomy prosection/demonstration previously described also assesses students' ability to use basic science in the practice of medicine. Students must research a clinical problem related to the anatomic structures they are dissecting, then explain to the faculty how the two relate. For example, during the dissection of the head, students describe surgical operations for strabismus, pointing out the ocular muscles involved in the repair, the muscle function, nerve supply, and origins and insertions.
Ability IV—Diagnosis, Management, and Prevention
At the beginning level, students acquire basic diagnostic skills while analyzing cases in PBL sessions in pathophysiology, particularly in regard to developing a differential diagnosis. In the introduction to clinical psychiatry course, students have an opportunity to diagnose patients' mental illness as they observe patients and participate in small-group seminars at hospitals and clinics.
Clinical evaluation of students is the standard approach on all the core clerkships and clinical electives. Students present history and physical examination information they have gleaned from a patient to the supervising resident or attending physicians. Students then generate a problem list and a plan for further investigation and management. These problem lists and plans are typically presented in both oral and written form, and feedback is given by the supervisors. The sum of these evaluations is used as the basis for a final grade, with more weight given to students' performances toward the end of the clerkship.
Oral examinations are used in the surgery clerkship in addition to the previously described clinical evaluations. The examination cases are drawn from a list of cases that appear on the students' logs of patients. Written examinations, prepared either by the faculty or by the National Board of Medical Examiners, are used in most core clerkships. A “key features” examination in obstetrics and gynecology assesses students' diagnostic and management skills. The test focuses on the critical steps in the diagnosis or treatment of a specific problem or condition. Questions can be tailored to a specific topic or concept by focusing on the steps considered most important to the decision-making process.6
Standardized patients in OSCEs are used to assess Ability IV in the medicine, obstetrics and gynecology, and pediatrics clerkships. Students are judged, for example, in terms of how well they identify risk factors and opportunities for preventive interventions. Students may also be asked to interact with the standardized patient to provide preventive counseling, especially in areas that require behavioral change such as smoking cessation.
Ability V—Lifelong Learning
The assessment of lifelong learning at the beginning and intermediate levels involves giving students tasks that require them to research, review, and extract information from multiple sources. Students are evaluated on how successful they are in finding the necessary information.
The degrees to which students possess the technical skills for lifelong learning can be assessed in PBL in which students report back to their small groups the results of their inquiries relative to specified learning objectives. In the anatomy prosection/demonstration, students are rated on lifelong learning based on the thoroughness of their literature reviews. In clinical clerkships, these skills are judged at the intermediate level by the appropriateness of the literature review that a student conducts when preparing a case presentation.
Ability V involves the internalization of an attitude of lifelong learning as well as possession of information-seeking skills. Faculty who have long-term relationships with students are in the best position to judge whether the students have internalized that attitude and have the proper skills. These faculty include preceptors for longitudinal ambulatory clerkships, research supervisors, career advisors, and other mentors who collaborate with students on research projects.
At the advanced level, students receive competency credit for exhibiting critical lifelong-learning skills during the research process. These skills include framing a question, utilizing sophisticated information-searching modalities, organizing data, and compiling and using information that results in a publication of an article, an abstract, or a poster or conference presentation. Over 50% of our students do substantive research and thus earn Ability V competency credit.
Ability VI—Self-awareness, Self-care, and Personal Growth
There is little doubt that traditional forms of assessment fall short in judging this ability. Yet, many faculty say that they are able to tell with a fair degree of accuracy those students who have a keenly developed sense of self and those who seem utterly clueless about how they come across to others. Some students actively seek out feedback, not to grub for a higher grade, but to truly learn from their mistakes, while others react to feedback defensively, blame others, and learn nothing. Some students deal with stress with equanimity while others fall apart. Some students manage to lead balanced lives with appropriate attention to their own physical, emotional, and spiritual fitness, while others become rigidly focused and unidimensional.
The faculty best situated to assess students are those mentors with whom the students interact over prolonged periods of time. In our curriculum, the academic advisors are in this position, serving as mentors over a four-year period beginning with the student's junior year in college. During the last two years of medical school, faculty career advisors are able to judge each student's ability to make good decisions and deal with the stresses of the residency matching process. Both of these sets of advisors assess and certify students for Ability VI at the beginning and intermediate levels. The clerkship in psychiatry also assesses and certifies students in Ability VI at the intermediate level (see the Appendix, first paragraph).
Elective Balint groups for students provide another opportunity for students to gain competency in this ability. The faculty supervisor assesses each student's ability to honestly reflect on his or her own reactions to clinical situations confronted daily and to use that information to improve clinical interactions and patient care.
Faculty can also nominate students for competency certification in Ability VI at all levels based on evaluative information that has been collected for other purposes. For example, a student was nominated for Ability VI certification based upon her journal entries during a service-learning placement in a hospice. The student wrote about her own father's death from cancer and the struggle that she had had as the end-of-life issues were poignantly reawakened when she went to the hospice for her coursework. By the end of the semester, her journals eloquently demonstrated the development that had occurred in her own self-awareness and personal growth.
Ability VII—The Social and Community Contexts of Health Care
In addressing the broader context of medicine, students need to be responsive to the many non-biological factors that influence health and to learn how to utilize community resources in order to be advocates for better patient and community health. Service-learning activities, where students work with individuals from diverse backgrounds, offer an excellent opportunity for students to demonstrate their competence in this ability. Students are assessed on their ability to accurately characterize the health of the community from an epidemiologic viewpoint and identify those factors, both biological and non-biological, that influence health and disease in that community. This is done through written reports, oral presentations, exhibitions of projects, and participation in reflection sessions, and by direct evaluations from community members and supervisors. The majority of students earn at least one beginning-level credit through participation in a student-led community service program. Students who play leadership roles in these organizations may also earn intermediate credit. An example of a rating sheet used to assess journal entries of students doing a service-learning project is included in Chart 1.
Clinical students are assessed directly for Ability VII at the intermediate level though the family medicine and community health clerkships as well as through those electives that use community-based projects. Course faculty and faculty mentors assess students through oral presentations using computer-generated slides, poster presentations, and/or patient write-ups, looking for evidence that the students have explored the social, cultural, psychological, economic, and spiritual dimensions of health care with patients and their families.
The Ability VII assessment committee appoints a faculty mentor to work with students interested in pursuing Ability VII at the advanced level. The committee directly evaluates students who seek advanced-level certification. These students are invited to the committee to make a formal presentation of their activities and engage in a seminar-style discussion with committee members. The committee ascertains the degree to which each student was involved in the conceptualization, planning, implementation, and analysis of a project focused on the health of populations, as well as the student's follow-through in terms of advocacy for the patient population.
Ability VIII—Moral Reasoning and Clinical Ethics
Competence in this ability requires that students recognize the ethical dimensions of medical practice and health policy and learn how to identify, analyze, and effectively carry out a course of action that takes account of the ethical complexity involved. (Unethical and unprofessional behavior on the part of the student, such as cheating, lying, stealing, falsifying records, and so on are dealt with through a disciplinary mechanism of the medical school and university.) Students earn their beginning-level competency by completing a foundation course in ethics.
At the intermediate level students' abilities to apply ethical analysis to the clinical setting are assessed through “ethics rounds” in the clerkships. At these rounds, a case is presented by one of the students to the other students and the bioethics faculty member. The presentation is followed by a group discussion of the case. A student earns credit toward competency certification by attending the ethics rounds and preparing a write-up of an ethics case in which he or she describes and analyzes the ethical issues.
Ethical sensitivity—the ability to recognize an ethical issue when confronted with it in context—is assessed in the fourth-year OSCE. Cases are structured in such a way that the student who does not possess ethical sensitivity could interpret the case as a straightforward biomedical problem without ethical components. Standardized patients are trained to rate students using the parameters described earlier.6List 3 includes a rating sheet used for these purposes.
Advanced credit may be obtained for completing an advanced group or individual project under the supervision of an appropriate mentor. These projects include an independent study in a clinical setting, an honors thesis or publication in bioethics, or an internship at a bioethics institute. The Ability VIII assessment committee is responsible for evaluating and approving the project for advanced credit.
Ability IX—Problem Solving
Ability IX involves the integration and synthesis of research and clinical data and the development of a plan of action in order to resolve a complex problem and implement a course of action. To successfully manage many patients in a complex organization such as a hospital, the student needs to demonstrate skills in juggling many competing demands, setting priorities, working efficiently, negotiating, advocating, and handling unanticipated obstacles.
Such problem solving can be assessed only in action, at least at more advanced levels. At the beginning level, Ability IX is assessed in a basic science course, such as human neurobiology, through weekly analysis of clinical vignettes and written examinations. At the intermediate level students are placed in a situation with responsibility for multiple complex tasks, then assessed in terms of success in accomplishing those tasks as efficiently as possible. Efficiency becomes increasingly more important as the student progresses in his or her professional development, since the number of multiple tasks to be handled concurrently will increase as the student becomes more experienced
The subinternship represents the epitome of such an assessment opportunity involving the clinical care of patients. In assessing problem solving at the advanced level during the subinternship, the focus is not only on individual patients that students are managing but also on how well they are managing all of their patients collectively. The faculty member assesses how well the students are able to set priorities, monitor their patients, be sure that investigations are being done expeditiously, communicate with the various health care teams, communicate with patients and their families, prepare for discharges, and arrange for appropriate community services and follow up.
As far back as 1951, Lindquist argued that “it should always be the fundamental goal of the achievement test constructor to make the elements of his test series as nearly equivalent to, or as much like, the elements of the criterion series as consequences of efficiency, comparability, economy, and expediency will permit.” In constructing tests, he adds, “the most important consideration is that the test questions require the examinee to do the same things, however complex, that he is required to do in the criterion situations.”7 [Italics in both quotes are Lindquist's.]
Our competency-based curriculum achieves these assessment goals by defining the tasks that our medical school graduates will be expected to do as physicians, then incorporating the knowledge, skills, abilities, and values that go into accomplishing these tasks into the medical school curriculum. In our courses, the faculty use multiple performance measures and mentoring over time to assess not only what the students know, but also their abilities to do what they know. The result is a curriculum that better captures the full range of what it means to be a competent physician.
1. Smith SR, Dollase RH. Planning, implementing and evaluating a competency-based curriculum. Med Teach. 1999;21:5–22.
2. Novack DH, Dubé C, Goldstein MG. Teaching medical interviewing. A basic course on interviewing and the physician—patient relationship. Arch Intern Med. 1992;152:1814–20.
3. Goslow GE. Human morphology at Brown (1990-1995): it's all in the team. Medicine & Health/Rhode Island. 1996;79(1):10–4.
4. Regehr G, Freeman R, Robb A, Missiha N, Heisy R. OSCE performance evaluations made by standardized patients: comparing checklist and global rating scores. Acad Med. 1999;74 (10 suppl):S135–S143.
5. De Gruijter DNM. Compromise models for establishing examination standards. J Educ Meas. 1985;22:263–9.
6. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med. 1995;70:194–201.
7. Smith SR, Balint JA, Krause KC, Moore-West M, Viles PH. Performance-based assessment of moral reasoning and ethical judgment among medical students. Acad Med. 1994;69:381–6.
Sample Courses Used for Evaluating Competence in Selected Abilities
Psychiatry Clerkship—Ability VI Intermediate Level
The six-week clerkship in psychiatry assesses students in Ability VI, “Self-awareness, Self-care and Personal Growth,” at the intermediate level. An attending psychiatrist meets weekly with students to guide them through a series of readings and live patient encounters. These experiences are designed to facilitate the students' reflections on their role as emerging physicians in relation to their patients, other health care workers, colleagues, and the system in which they work. Students are also afforded the opportunity to examine their own coping styles and to think about ways to take care of themselves while managing the stresses of their medical training.
Subinternship in Medicine—Ability IX Advanced Level
Ability IX—Problem Solving is assessed at the advanced level in the subinternship in medicine based upon the student's executive ability to “get the job done.” Faculty pay particular attention to how well a student can juggle multiple tasks simultaneously in an environment that is often chaotic and high-pressured. Priority setting, negotiating, and time-management skills play paramount roles. Flexability, equanimity under pressure, perserverance, and stamina reflect the personal and professional characteristics needed for success. Subinternship faculty also assess the more traditional abilities of basic clinical skills, and diagnosis, management, and prevention through clinical observation at the bedside, and review of students' write-ups.
Human Neurobiology—Ability IX Beginning Level
The first-year neurobiology course addresses Ability IX—Problem Solving, by helping students deal with the practical problem-solving process that they will face as future physicians, confronting patterns of abnormal neurologic findings from which they must infer the structures affected and the locations of the pathologic processes. Each week, students work through case vignettes that help them identify patterns of common neurologic disorders. To aid students in this complex problem-solving process, the course leader provides an analytic framework that consists of a series of questions that allow students to narrow the possibilities systematically. Evaluation of this competency is assessed through small-group discussions and problem-solving questions on the written examination.
Human Morphology—Abilities I, III, and V Beginning Level
Human morphology (anatomy) faculty utilize an innovative assessment method called prosection/demonstration to assess Ability I—Effective Communication, Ability III—Using Basic Science in the Practice of Medicine, and Ability V—Lifelong Learning. Students research clinical applications related to the regions they are dissecting, then present their findings, including the sources used, to the basic science faculty and visiting clinicians in ten-minute presentations at the dissecting table. Each presentation is followed by a five-minute question-and-answer interchange with the faculty. In addition to being judged on his or her knowledge of anatomy, the faculty assess the student's ability to use library resources, to apply basic science knowledge of anatomy to clinical situations, and to effectively communicate all of this to professional colleagues.
Pathophysiology—Abilities I, IV, V Beginning Level, Ability III Intermediate Level
The second-year pathophysiology course assesses four abilities: Ability I—Effective Communication, Ability IV—Diagnosis, Management, and Prevention, Ability V—Lifelong Learning at the beginning levels, and Ability III—Using Basic Science in the Practice of Medicine at the intermediate level. Students meet in problem-based learning groups six hours a week. In these sessions students are encouraged to communicate clearly, participate in different roles (e.g., listener, leader), and respond sensitively and respectfully to others. Small-group faculty facilitators rate students on their competence in effective communication as well as the other abilities using a behaviorally anchored rating scale. Facilitators also provide feedback to individual students at the midpoint and end of each pathophysiology section.
Family Medicine Clerkship—Ability VII Intermediate Level
The family medicine clerkship requires every third-year student to demonstrate competency at the intermediate level of Ability VII—The Social and Community Contexts of Health Care by completing a community-based project. Students choose a patient-focused topic or project that interests them. They are then matched to a community preceptor site based on their clinical interests. At the end of the six-week clerkship each student makes a PowerPoint presentation to the class and faculty. Students also typically produce a resource sheet for use by their preceptors. Examples of projects include Intervention for Diabetics in a Portuguese Community and Medical Students Outreach to Mothers-to-Be.
8. Lindquist EF. Preliminary considerations in objective test construction. In: Lindquist EF (ed). Educational Measurement. Washington, DC: American Council on Education, 1951:119–84.