Kalet, Adina L. MD, MPH; Sanger, Joseph MD; Chase, Julie PhD; Keller, Allen MD; Schwartz, Mark D. MD; Fishman, Miriam L. MD; Garfall, Alfred L.; Kitay, Alison MD
By three methods we may learn wisdom: first by reflection, which is noblest; second, by imitation, which is easiest; third by experience, which is bitterest. - —Confucius, Confucius: The Analects
As medical educators we are being challenged to assess the professionalism of our students.1,2 Recent evidence linking certain unprofessional behaviors in medical school with subsequent disciplinary action in practice reinforces that professionalism assessment should be part of the medical profession’s contract with society.3 The emphasis on assessing professionalism makes educational sense, since assessment motivates students to learn what is important and gives the faculty valuable feedback on the effectiveness of the curriculum. The issue remains controversial, however, because although there is agreement on the principles comprising professionalism,4 there is less consensus on a behaviorally specific definition of professionalism, and there are few reliable and valid tools for measuring professionalism.5 Furthermore, there is reluctance on the part of educators and peers to judge a student’s professionalism in high-stakes situations, such as being unwilling to identify fellow students cheating on an exam.6,7 For all these reasons, it is essential to develop explicit, meaningful, developmentally oriented, and structured approaches to promoting the growth of professionalism in our trainees.
Arnold and Stern8 have suggested the following definition of professionalism:
Professionalism is demonstrated through a foundation of clinical competence, communication skill, and ethical and legal understanding, upon which is built the aspiration to and wise application of the principles of professionalism: excellence, humanism, accountability, and altruism.
This definition suggests that clinical competence, high ethical standards, legal knowledge, and communication skills are necessary but not all-encompassing elements of professionalism. In addition students need to demonstrate the wise application of and commitment to the loftier ideals of professionalism.
Professionalism in the Curriculum
Why formally teach professionalism?
Students entering medical school are idealistic but have only a general and superficial understanding of the values and characteristics that define medical professionalism. In a survey we conducted of the entering New York University (NYU) School of Medicine classes of 2004–2006, there was a high level of agreement among students that physicians were expected to be honest and compassionate, have excellent communication skills, and strive for excellence, but participants were less able to describe specific professional or unprofessional behaviors.9 There is long-standing evidence that medical training erodes idealism as students develop attitudes that may negatively impact on patient care. For instance, students become more cynical and self-report more unethical behavior from first to fourth year of medical school,10,11 and some students develop negative attitudes toward certain groups of patients (e.g., the elderly, HIV-infected patients).12,13 The ability to demonstrate patient-centered communication skills has been shown to deteriorate during medical school.14,15 Therefore, given that students come to us idealistic and with high expectations of themselves, our job is to ensure that the learning environment promotes those core values and the skills to uphold them in the face of challenges. For instance, students need to recognize and analyze conflicts between the values of professionalism and the daily pressures of practice and training, thus embracing the privileges of a self-regulating profession and the obligation to set and maintain standards of competence and integrity.4,16,17
In a recent critique of the current debate on medical professionalism, Jack Coulehan18 proposed a narrative-based approach to help trainees address the tension between self-interest and altruism. This approach would emphasize role modeling, self-awareness, narrative competence, and community service, all of which would form the basis for written reflection by the trainee that could be captured in an educational portfolio. Critical self-reflection of this kind, when reviewed with a faculty mentor, has the potential to ensure that the critical professional values are learned.
A portfolio is a purposeful collection of student work that exhibits the student’s efforts and progress in selected domains. As an assessment tool, portfolios provide direct evidence of the quality of a student’s work and accomplishments, and they nurture the ability to readily reflect on one’s experiences as is needed in professional work of all types.19 Portfolio learning is used in a wide range of medical training settings because it encourages self-directed learning, fosters learning about how one learns (metacognition), and demonstrates progress toward identified outcomes.20–22 Not only has the use of portfolios been shown in a randomized trial to increase learning in undergraduate medical students,23 but this approach is appealing because it respects individuality and diversity. It also has the advantage of generating material for authentic and personalized assessment while also developing lifelong learning skills by emphasizing the process as well as the products of learning.
Adult learning is generally more effective when it is goal oriented, learner centered, and experiential. Experiential learning is a cyclical process of concrete personal experience, reflection, abstraction, generalization, and testing of implications. Reflection translates the experience of clinical practice into learning and is a crucial intellectual task in professional competency.24
At NYU School of Medicine, we collaborated with student members of the Professional Development Committee, medical educators, medical information technologists, and faculty mentors to develop, implement, and evaluate the Professional Development Portfolio (PDP). The PDP is a Web-based, password-protected application designed to support students’ professional growth and development and reinforce professional values and beliefs. Students regularly upload material they produced as part of the required medical school curriculum and write a few sentences of reflection on how participating in a given exercise affected their professional development (e.g., feedback from a standardized patient encounter, a paper integrating the molecular biology of genetic disease with its clinical manifestations and impact on the patient). Students meet annually with faculty mentors to review their reflections, assess their professional growth, and set goals for the following year. In this article we describe the PDP, share our experience with its implementation, and present initial student views on this novel project.
Design of portfolio
The assumptions underlying the PDP’s design are as follows:
▪ The development of professional identity and character is a dynamic and multifaceted process.
▪ Healthy professional development and appropriate behavior require continuous and honest self-assessment and the ability to reflect thoughtfully on the complex and challenging situations regularly encountered in professional practice.
▪ A student’s professionalism is expressed in every educational venue and not just in clinical contact with patients.
Each student has a PDP that can only be viewed by the student, the assigned faculty partner, and the PDP administrator. The matrix shown in Figure 1 is a screenshot of the front page of an individual student’s portfolio, which functions as a table of contents.
We established the goals of the PDP (List 1) and, in discussions with students and course directors, identified at least one concrete educational activity from each of the required courses in the preclinical curriculum through which students were likely to manifest their professionalism. For example, the director of the Human Anatomy course believed that students are most likely to express their professionalism through their interactions with peers around the cadaver dissection table. We designed a peer feedback activity in which every student gives written feedback (structured responses and open text) to each other member of the dissection team. After receiving feedback from peers, students were asked to write two to three sentences in their PDP reflecting on what they learned from participating in this exercise. Other required submissions include essays on the physician–patient relationship, evaluations from standardized patient interactions, and documentation of other assessments from faculty and peers conducted in the course of the preclinical and clinical curricula. There are also opportunities for students to submit individualized elective materials, such as documentation of community service projects and elective seminars. After uploading the relevant documents, students write a brief reflection on how the activity affected their professional development.
Toward the end of each year, students are reminded by e-mail to write a one to two page summary of their professional development during that year, write learning goals for the coming year based on that summary, and assess their own progress by giving themselves grades. Students grade their professional development on a three-tiered scale: falls below expectations, meets expectations, or exceeds expectations. The evaluation criteria (List 2) emphasize thoughtful participation in the process but do not explicitly require a judgment of the quality of students’ submissions or reflections. In this way, we reinforce that we are defining professionalism as a process rather than a specific episode of behavior.
We designed a “comment card” function in the portfolio that allows others to post written remarks, either praise or concerns, to a student’s PDP (Figure 2). Any member of our student body, faculty, or staff with e-mail access can log onto the publicly available front page of the PDP Web site and file a comment card on any student with a PDP. Comment cards are not anonymous. To file a card a person needs to identify his or her relationship to the student, is then prompted to discuss the issue being raised with the student, and only then are they asked to describe the issue in detail in the text box provided. Once submitted, the card is automatically sent to the student, the faculty mentor, and the PDP administrator. If remediation is required, the faculty partner activates the investigatory or disciplinary procedures of the dean’s office. Faculty partners are repeatedly reminded by e-mail until they review the card and enter a note, in a text box, describing action taken. Once the faculty partner submits comments, the card is identified as closed and appears on the student’s matrix in a faded color.
External uses of submitted materials
Student participation in the PDP is required. Students may elect to submit material from the PDP to the dean, who then may draw on its contents in composing the dean’s letter or to consider a student for graduation honors or Alpha Omega Alpha honor society membership. Only a few students have taken advantage of this option. Any student who stays in our medical center for residency training and beyond may continue to access their PDP. Arrangements are underway to ensure ongoing access for all graduates.
The PDP was designed as a Web application to allow wide accessibility and easy updating using software with which we have expertise and experience. An Oracle database was employed because it is a standards-compliant, robust, and scalable relational database that offers great stability, easy Web connectivity, and a multitude of available third-party support tools. We used the Fusebox development methodology, layered on top of the ColdFusion language, to provide an overall structure to our application’s design. In the PDP, we have elected to store text data within Oracle itself, while uploaded documents (text files, Microsoft Word and PowerPoint documents, and Adobe Acrobat PDF files) are stored within the standard file storage directories of our Web server; only pointers to these documents are stored in the database.
We first implemented the PDP in the 2003–2004 academic years, with the entering class of 2007. Currently, all 720 NYU medical students maintain a PDP. Compliance with all aspects of the PDP has been high, with only two students in the class of 2007 not completing all required PDP submissions. Approximately 20 comment cards per class are filed each year (81 cards on 62 students during four years). Most are filed by classmates, student club leadership, or student government representatives, and all but three have been detailed descriptions of praiseworthy behavior. Of the three critiques, the most high-profile was filed by a member of the dean’s office administrative staff regarding a student who, out of anger, sent a disrespectful and demanding e-mail, copied to the entire student body. The comment card immediately triggered a meeting between the student and mentor, and the issue was resolved without delay.
We made it a high priority to resolve all technical and logistical challenges, which were greatest in the first year, as expediently as possible, and we have since attended to adjusting the requirements to enhance the educational value of the PDP. The PDP is reviewed annually, and changes are made with extensive student involvement and faculty input.
Training of faculty partners
On entry to NYU School of Medicine, all students select affiliation with one of five theme-based Master Scholars Program (MSP) Societies within which they are assigned to a mentoring group of two faculty mentors and eight students. These mentoring groups meet seven times each year in the preclinical years, and each student meets individually with one of the two mentors twice annually. The final student–mentor meeting of each year is devoted to conducting the end-of-year assessment for the PDP. We have recruited 80 faculty mentors from all preclinical and clinical departments to run the mentoring program and be faculty partners for the PDP. Mentors are selected for their reputations as teachers and/or clinical role models.
In parallel with the implementation of the PDP, we conducted a series of faculty development sessions aimed at introducing faculty mentors to the PDP. The specific goals of the sessions were to introduce the adult learning principles underlying the PDP and to ensure that the faculty could recognize professionalism issues, give effective feedback, facilitate a self-assessment, craft good learning objectives, and negotiate a grade. Most faculty partners participated in two, one-hour sessions that included role playing with students in mock end-of-year assessment sessions.
Assessing Student Views of the PDP
In addition to regular “town hall” meetings to discuss student portfolios at the end of the academic year, we administered a 17-item survey to elicit students’ views on the usefulness and effectiveness of the PDP, its ease of use, and the students’ experience of technical difficulties. Items were generated to assess whether we had accomplished our goals and to identify areas that could be improved. The NYU School of Medicine institutional review board has reviewed this survey. Data were collected anonymously through a Web-based survey tool independent of the PDP.
We summarized responses for 15 of the 17 survey questions for which the responses were on a Likert scale of one (strongly disagree) to five (strongly agree) to show the proportion of students who agreed (one or two) or disagreed (four or five) with statements about the PDP. We assessed for differences in responses among subsets (a student’s class year, or whether a student experienced technical difficulty) using chi-square tests. Narrative responses were read and themes identified independently by three students (M.F., A.K., A.G.) and one faculty member (A.S.K.). Through an iterative consensus-building process we arrived at a set of themes that best described the data. Another faculty member (A.L.K.) applied these themes to the data and summarized the results for review by the team.
Table 1 presents the responses of the 164 students to the 15 closed-ended survey items administered at the end of the first year for the class of 2008 (response rate 74/160, 45%) and end of the second year for the class of 2007 (response rate 90/160, 56%). These results present a mixed view of student satisfaction with the PDP. For example, a plurality of students felt the PDP was a useful way of tracking professional development and a reasonable way to demonstrate professionalism. A plurality also disagreed, however, that the self-assessment process increased awareness of professional responsibilities. A majority found creating the portfolio challenging. Revealing a significant opportunity for improvement, a majority found their faculty mentors unprepared for the end-of-year meeting. There were no significant differences in survey responses between the classes, and there were no differences in survey responses between the 48% (79/164) of students who reported experiencing significant technical problems with the PDP and those who did not.
Students were asked specifically to list the aspects of the PDP that were either helpful or not helpful. Of the 72% (118/164) of students who provided narrative responses, many responded to both questions or combined their answers in one textbox. We therefore analyzed all 156 unique comments, 102 of which were positive, as one data set. List 3 summarizes the themes in the student comments with representative quotations.
Evolution of the PDP Over Four Years
The Professionalism Development Committee consists of 25 students, two faculty advisors, and a project manager who meet every one to two months to address issues raised about the PDP. At the conclusion of each academic year the PDP has undergone extensive review by students, faculty, and administrators. Significant changes have been made including adding more specific instructions for each of the submissions, eliminating some requirements, and making the requirements more flexible (e.g., allowing students to submit reflections for just one of three standardized patient encounters rather than having to reflect on all three). As the first class of participating students matriculated to the clinical years, submissions were devised for each of the seven required clerkships in collaboration with clerkship directors. For example, on the obstetrics–gynecology and psychiatry clerkships, students who worked closely together exchange feedback and reflect in their portfolio on insights made by their colleagues. Submission requirements are continually revised based on student feedback, changes to the curriculum, and specific events in our school that prompt discussion of particular professionalism values.
More than 80 faculty partners have been trained, and there is now interest in creating a similar PDP for some of our GME programs.
Technical issues, which have largely been addressed, include (1) session “hijacking,” where students have gained access to another’s portfolio, most likely because the student had accidentally forgotten to log off a computer in a public area, (2) data loss attributable to an automatic log-off feature, and (3) incompatibilities attributable to the use of multiple operating systems by students.
Funding of the PDP
Faculty time for the initial three years of development of the PDP, including Web application design and programming and evaluation, was supported through a series of small grants (American Medical Association, Strategies in Teaching and Evaluating Ethics and Professionalism program, the American Board of Internal Medicine Project Professionalism Foundation, and the Mannix Medical Education Award New York State Medical Society). Currently, the entire project, including Web application maintenance and administrative and faculty support, are funded through the dean’s office.
Summary and Discussion
We created and implemented the PDP, a password-protected, Web-based application that allows students to collect evidence of and narrative reflections on their own professional development, which is reviewed annually with a faculty mentor. This is a self-assessment tool that also allows faculty and peers to submit feedback to students about how the students have demonstrated their professionalism. In this way, we created a method of assessment of professional behaviors that is potentially longitudinal, dynamic, and meaningful. By focusing assessment on concrete examples of a student’s work, such as an SP interaction which requires the student to negotiate an STD prevention plan with a teenager who doesn’t want her parents to know she is sexually active (demonstrating issues of honesty, respect, and integrity) or reflecting on feedback received from a peer (demonstrating commitment to excellence and accountability), we steer discussion away from character traits such as amicability and politeness and toward developing lifelong habits associated with maintaining professional behaviors in the face of increasingly complex situations. Most students took the PDP very seriously and, either with enthusiasm or begrudgingly, accepted and met the expectations set out for them. Also, it is our experience that most hold themselves to very high standards of excellence, responsibility, and integrity toward their peers and teachers.
We chose to base our assessment of a student’s professionalism on his or her participation in the process of maintaining a portfolio, writing reflections on his or her own development, and participating in peer- and self-assessment. Our rationale is that a physician’s professionalism is not a stable set of traits but, rather, a set of decision-making skills practiced in challenging and often ambiguous situations. Evidence supports that professionalism is best examined when students make context-specific decisions in the face of conflict between and among the various tenets of professionalism. It follows that students must learn these tenets, understand the common scenarios in which professionalism is challenged, and be aware of options for behavior.25 Therefore, one’s professionalism overall is a matter of personal identity and character that is shaped by clear expectations, knowledge, skills, and experiences. Although professionalism is most obviously tested during periods of stress, it is more commonly manifested during routine activity. Future physicians must be committed to developing and refining the requisite wisdom to know what to do when professionalism is challenged.1 Ideally, the PDP supports this development in our students.
Student satisfaction and belief in the utility of the PDP is mixed. Many students think the PDP is useful and reasonable, even though most feel the tasks involved in maintaining it are not easy. Opinions are most heterogeneous regarding whether the PDP contributed to professional development and made mentoring sessions more useful. Some students are convinced that this is a rote exercise that is unlikely to lead to sincere, meaningful self-assessment. Others feel writing requirements not focused on biomedical or technical information are “flaky” or “touchy feely,” and others find only the end-of-year summary useful. On the other hand, some ask for more frequent, briefer entries that resemble a personal journal. A small group of students just want to be left alone until they become clinically active, and others feel that, at their age, it is too late to impact professional development. Despite this heterogeneous response to the particulars of the PDP, many students expressed appreciation that the school was emphasizing professionalism. These complex responses by preclinical students to the PDP suggest avenues for improvement of the PDP through individualization and tailoring of requirements. In addition, we believe this is also evidence of the PDP’s utility as a means for encouraging personal introspection and lively public discussion of complex issues. Moreover, we expect that some students who are averse to the PDP in the early years of their training may later find it valuable, and we suspect that our approach may be most beneficial for students who are initially reluctant to participate in this type of exercise but nonetheless open-minded. Therefore, student satisfaction is an important, but not the sole, measure of the PDP’s success.
Although it is true that, for some students, written exercises are not the preferred or most comfortable means of expression, most educators would agree that this does not negate the importance of being able to write articulately about one’s own views on issues related to professionalism. We plan to examine these subtleties in students’ learning styles further and in so doing perhaps devise a collection of individually tailored professionalism assessment strategies to allow students to achieve the goal of professional development through the most effective means.
Anecdotally we are finding that students who have maintained a PDP are more articulate about professionalism issues in the clerkships, but the PDP’s impact on actual behavior will not be clear for some time. We are hopeful that, because the PDP was designed with substantial student input and is revised annually, ultimately a positive impact will be demonstrable. A study of the PDP’s impact on students’ attitudes and beliefs about professionalism and their moral reasoning is ongoing.
The majority of students did not experience their faculty mentors as prepared to use the PDP in mentoring sessions. This suggests that students expect mentors to come to meetings familiar with the contents of their PDP. Our faculty development efforts involved a single, two-hour workshop and focused on the conduct of the end-of-year assessment. More detailed training is probably necessary for faculty mentors to be more engaged with the PDP and more skillful in its use with students.
The existence of the PDP at NYU has increased the demand from students for symmetry in evaluation. This idea, currently embraced in the modern business and organizational management arenas, that all members of an organization should assess each other nonhierarchically using the same processes has yet to be broadly accepted in our medical school. However, we are now discussing the creation of a similar professionalism assessment of residents and faculty.
Other schools have mature and successful critical incident-reporting systems or early warning systems to identify students demonstrating unprofessional behaviors.3 We hope that the comment card in our PDP, intentionally labeled neutrally (i.e., termed a comment card rather than a concern card or warning card) to encourage students and faculty to submit positive examples as well as critiques, will enhance the total amount and quality of the feedback our students and faculty give to each other and therefore create a culture in our school where minor lapses in professionalism can be addressed early.
We have learned a great deal from our experiences with the PDP. We agree with others that a Web-based portfolio system can greatly facilitate learning and assessment of professionalism, a critical component of a physician’s competence that is difficult to assess by conventional means.20 We have demonstrated that such a system is workable, and, if implemented through a respectful, responsive, and open process, is generally acceptable to students and capable of stimulating thoughtful discussion and meaningful assessment of professionalism. We believe that self-assessment of professionalism, in the context of a mentoring relationship and a narrative portfolio that allows students to document their professional development actively, offers significant advantages over quantitative assessments of professionalism by faculty, which are historically unreliable.5 More important, a system like ours places responsibility for professionalism assessment in the hands of students, thus serving as a proving ground for their future roles as independent practitioners in a self-regulated profession.
The authors would like to thank Corry McKee and Lisa Manigo for vigorous administrative support, then-Senior Associate Dean of Student Affairs Mariano Rey for spiritual support, now Assistant Dean of Student Affairs Lynn Buckvar-Keltz and the many student leaders of the PDC for being such inspiring colleagues in this process. This project was funded in part by AMA STEP Program, ABIM Project Professionalism Foundation, and the Mannix Award of the NYS Medical Society.