Secondary Logo

Journal Logo

Assessment Reconsidered: Finding the Balance Between Patient Safety, Student Ranking, and Feedback for Improved Learning

Sklar, David P., MD

doi: 10.1097/ACM.0000000000001687
From the Editor
Free
SDC

Editor’s Note: The opinions expressed in this editorial do not necessarily reflect the opinions of the AAMC or its members.

As I looked up from my computer, I noticed that Tina (not her real name), one of our interns, was standing by my doorway. Tina’s eyes were bloodshot, and it looked as though she had been crying. “Come on in,” I said. “Is there something wrong?”

“I just read my evaluation from you, from our shift last week. I’ve never had such a bad evaluation before.”

“But it wasn’t a bad evaluation,” I said. “It was a good evaluation. I don’t understand. You managed all of your cases well. I checked the boxes that said ‘meets expectations or exceeds expectations’ for every competency. Why do you think it was a bad evaluation?”

“In the comments it says that I need to improve my communication skills. No one has ever accused me of that before.”

“Tina, I try to identify areas of strength and areas where there could be improvement for everyone,” I said. “We can all improve our communication skills. I mentioned it because of the woman you were managing who had the rash and left against medical advice. She was a very difficult patient; I think you probably didn’t realize that some of your suggestions about her need to lose weight and to address her poor hygiene came across in a way that you didn’t intend. I sense that she felt judged and stigmatized, and that was why she stormed out. I wanted to encourage you to work on how to communicate about sensitive issues with patients like that lady. But you did nothing wrong. We talked about this at the end of the shift. I wrote this in my comments to remind you.”

“But the woman was living in her car and had scabies. The scabies were the result of poor hygiene, so she should bathe every day. She has to go to a shelter and lose weight. Her weight problem probably has led to her diabetes and high blood pressure. I explained all this to her as clearly as I could. Yet you wrote that I need to improve my communication skills; that statement will be in my record when I get evaluated by the program director and when I apply for a fellowship.”

“Tina, I’m sure my comments will be viewed in the way they were intended: as feedback for you. We physicians have to be aware of how we communicate. We must understand the lives and challenges of our patients and how to establish rapport and trust. Communication is not just about transfer of information. If I didn’t write up comments, there would be no record that I was supervising you and doing a workplace-based assessment of your progress with the milestones and competencies. We need to document that for everyone. Look, this is a fine evaluation. I’ll speak with the program director and clarify my comments.”

Tina rubbed her eyes, took one more distrustful look at me, and then left. I looked back at the forms I had filled out assessing her and other residents and wondered whether Tina’s fears could be justified and whether she would be labeled in a negative way based on my comments. Even though my feedback had been given with the intent of helping Tina with her communication skills, I worried that there could be unintended consequences.

I decided to learn more about assessment so I could find out what the literature suggests about how best to navigate the difficult challenge between providing an assessment that would lead to improvements in learning, and implying a judgment about attitudes or performance that might end up in a student’s evaluation.

Back to Top | Article Outline

Goals and Challenges of Assessment

Assessment is needed to help students improve, but it is also necessary to identify students with significant deficiencies and to assist in the sorting of students for selection to residencies, fellowships, or independent employment as physicians. How can assessment meet these conflicting goals when it seems to depend on a trust between faculty member and student that could be endangered by the power differential between them?

Van der Vleuten1 has described five criteria to be considered in assessment: reliability, validity, educational impact, acceptability, and cost. The workplace-based assessment that I had been doing with Tina, while potentially having educational impact, incurred high cost (of faculty time) and had relatively low validity and reliability as a single assessment. And, at least from Tina’s point of view, part of my assessment was not acceptable. The narrative comments, while useful to expand upon the ratings I marked, might not have much independent value as a predictor of future problems.

In this issue of Academic Medicine, Hatala et al,2 in their systematic review of qualitative comments from in-training evaluation reports (ITERs), found that narrative comments supported assessment of performance based on other types of quantitative scoring, but that evidence was lacking for making decisions or drawing implications. Whether one could extrapolate from the ITER to the end-of-shift evaluation form that I filled out is not clear, but the concerns about the limited validity of narrative comments are important to consider.

Kennedy et al3 have suggested that supervisors in the clinical environment should make an assessment about the trustworthiness of their students for independent clinical work that involves the dimensions of knowledge and skills, discernment of limitations, truthfulness, and conscientiousness. In making assessments, it may be useful to include comments about these trustworthiness dimensions in narrative comments, since they appear to be a part of the supervisory relationship and can be incorporated into the assessment of entrustable professional activities (EPAs).

Epstein4 provided an overview of assessment that described its use as part of the developmental model of competence in medical education as the student moves from novice to expert.5 He discussed common assessment methods such as written examinations; assessment as part of supervision by clinicians; direct observation and video review of specific clinical encounters; clinical simulations; multisource assessments by peers, nurses, and others; and portfolios. He recommended that the

various domains of competence should be assessed in an integrated, coherent, and longitudinal fashion with the use of multiple methods and provision of frequent and constructive feedback.

For the journey from novice to expert, Epstein said that a specific focus on the development of expertise is needed, and this requires assessment and feedback from teachers to learners.

Ericsson6 has described how teachers help students become self-directed learners by providing the feedback about performance that leads to deliberate practice to develop expertise. Thus, the role of assessment in the improvement of students’ learning is critical. However, it is important to recognize that there are several purposes for assessment, mentioned below, and that these can come into conflict.

Van der Vleuten et al7 recently summarized a model of assessment that divides assessment into three categories: assessment for learning, assessment of learning, and assessment as learning.

In assessment for learning, often referred to as formative assessment, students

are engaged in a process that focuses on metacognitive strategies … are supported in their efforts to think about their own thinking … understand the relationship between their performance, their current understanding, and clearly defined success criteria, and are positioned as the agent improving and initiating their own learning.

Konopasek et al8 recently described the importance and challenges of the formative assessment process in medical education.

Students often come to medical school from a culture that focuses on individual achievement and competitive advantage, and they may not be prepared to accept constructive feedback or to trust that anything but positive feedback will somehow put them at a competitive disadvantage.

They go on to note that “an assessment system that emphasizes the formative can only thrive in a culture that embraces and supports improvement.” They suggest the need for faculty development, learner development, longitudinal academic advising and coaching, and documentation of developing competence.

Assessment of learning, often referred to as summative assessment, emphasizes a judgment about a student’s current progress toward competency. While formative and summative assessments are often considered as separate goals of assessment systems, there is often overlap. For example, if an assessment meant to be formative detects serious deficiencies, it may lead to a judgment about the need for remediation or even dismissal from the training program. Similarly, a summative assessment may provide memorable experiences that motivate learning for the student. Turner et al9 in this issue describe summative assessment using pediatrics milestones and competencies. They demonstrate that through the use of milestones, trained faculty were able to distinguish the progress of interns from that of fourth-year students. Interestingly, there was little difference between interns and students in their attainment of the professionalism milestone, raising questions about the discriminating ability of the current pediatrics milestones for professionalism.

The third part of Van der Vleuten’s model, assessment as learning, incorporates the programmatic and curricular implications of the assessment process that might involve looking back from the assessment system to the curricular experiences and the goals of the educational program. Assessment as learning might also consider incorporating the effects of relationships between assessors and students, such as what occurs in the entrustment process.

In this issue, Lomis et al10 and Brown et al11 describe a framework for assessment of medical students that may be viewed through the lens of the Van der Vleuten model. Beginning with a description of the development of EPAs for graduating medical students, Lomis et al go on to describe the concept of entrustment, the assessment system, the curriculum, and faculty development. Brown et al further describe how summative entrustment decisions about core EPAs can be made by a trained group incorporating longitudinal workplace-based assessments that require evidence that students are truthful, conscientious, and know their limits, which are categories derived from the work of Kennedy et al.3 Brown et al note that the success of the assessment of these Core EPAs for Entering Residency will be dependent on the alignment of the curriculum, the assessment system, the goals of the program, and the development of faculty, and will involve formative and summative assessments and program evaluation.

While these suggestions for assessment make sense, they do not adequately address the conflict in the assessment system between its summative use for the ranking and selection of students for medical school, residency, fellowship, and employment, and its formative use to provide feedback needed for learning. The consequences of the assessment of learning influence the motivation of students about what to learn and how to prioritize their time on learning as well as their relationships with and trust of faculty supervisors. In addition, because the assessment system heavily influences learning, the content and processes of assessment of learning must be well aligned with the goals of the medical education system, or the physicians whom that system produces may not be properly trained to provide high-quality care.

Back to Top | Article Outline

Five Suggestions

These concerns lead me to make the following five suggestions.

The purpose of medical education must be well articulated and aligned with the assessment of learning. It is not sufficient to assert that the purpose of medical education is to prepare the student for the next stage of education, because learning is a continuum and there must be an overall vision for its ultimate goals. Also, preparation for subsequent stages in education is important but insufficient as a goal because of the many directions that students can take after each stage in their education. Such an approach abdicates responsibility for the design of the education system to meet the needs of the health delivery system and the population needs of improving health.

Without a clear consensus of the purpose of medical education, it becomes difficult to prioritize the ever-increasing content areas of health and health care. In his Invited Commentary in this issue, ten Cate12 connects the assessment of entrustment decisions to the purpose of medical care, which he identifies as patient safety. In this way, assessment becomes intertwined with faculty responsibilities to the patient for safe and high-quality care. Sklar and Lee13 have previously suggested that the purpose of medical education is high-quality care and used the six attributes of quality identified by the Institute of Medicine14—safe, timely, effective, efficient, equitable, and patient-centered—to help guide curriculum development and prioritization.

Whatever purpose for medical education one chooses, there must be a general agreement and alignment with the assessment goals. Without an overarching purpose for medical education, committees of experts brought together to define assessment content will tend to identify those topics and concepts that reflect their biases and experience, and may come into conflict with others who have different biases and experiences, leading to decisions based on influence and power rather than a clear vision. This may result in the inclusion of testing materials that could be peripheral to the core of knowledge and skills needed for health professionals. Assessment tools should be continuously examined for validity and reliability as the goals for the health professions change. A process that brings together accreditors of programs, assessors of competence, educators, researchers, practicing providers, and representatives of the public should be involved in reviewing goals, core content, and skills on a regular basis.

The effects of the assessment of learning (summative assessment) on the learners’ selection for education and employment, and for certification and licensure, should be acknowledged and managed. This should be done so that summative assessment does not adversely affect the assessment for learning (formative assessment) that is so critical for the education process.

This is more easily said than done, because there are pressures on the assessment system to help training programs to rank and filter students, who outnumber the available training positions. The assessment system can inadvertently facilitate mistreatment of students and residents because of their vulnerability in the assessment process and the imbalance of power between them and faculty. The guiding principles of assessment of learning should emphasize fairness, validity, reliability, authentic longitudinal relationships between faculty and students, entrustment, demonstration of competence, and integration of competencies—all in the service of quality patient care. There should be transparency and accountability for the assessment processes.

Assessment for learning (formative assessment) should be the foundation of the assessment program. Such assessment should include multiple sources of feedback, emphasis on self-directed learning, development of trusting and longitudinal relationships between faculty and students, and clear guidelines about when formative assessment may be used to influence promotion and other structural decisions. By emphasizing trust in the assessment process, it is less likely that students will be at risk for misinterpretation of a constructive comment. It is perfectly understandable that a student who feared that a critical comment on an assessment could adversely affect future training opportunities would not want the comment to appear in a portfolio, as in the case of Tina presented earlier. If such comments are meant to be formative, they should be shielded from summative review unless they suggest an egregious problem requiring immediate attention.

Assessment should be recognized for the critical role it has in overall educational program development. Assessment has major effects on the learning environment, curriculum development, relationships between learners and teachers, and institutional culture. While expertise in assessment in education is important, assessment should not occur in isolation; it must be integrated and aligned with programmatic goals and objectives.

Faculty development in assessment should be a core competency for faculty, particularly clinician educators, who will provide much of the workplace-based assessment. Favreau et al15 in this issue introduce key skills for faculty development as part of the assessment of EPAs. They suggest that faculty development include (1) training in observation skills in authentic settings (workplace-based assessments), (2) feedback and coaching skills, (3) self-assessment and reflection skills, and (4) peer guidance skills developed through a community of practice. Turner et al9 similarly identified the importance of faculty development as part of their study of milestone assessment in pediatrics. With the evolution of competency-based education, milestones, and the spread of EPAs in many programs, faculty may become confused with the changing terminology and concepts of assessment and lose site of the purpose.

Back to Top | Article Outline

Our Responsibility

The purpose of assessment is to foster safe, high-quality care to our patients, and if we can keep that in mind, the assessment system will make sense. It is our responsibility to make sure that assessment is understood, valued, and integrated into the fabric and culture of our health education institutions.

David P. Sklar, MD

Back to Top | Article Outline

References

1. Van Der Vleuten CPThe assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1:41–67.
2. Hatala R, Sawatsky AP, Dudek N, Ginsburg S, Cook DAUsing in-training evaluation report (ITER) qualitative comments to assess medical students and residents: A systematic review. Acad Med. 2017;92:868–879.
3. Kennedy TJ, Regehr G, Baker GR, Lingard LPoint-of-care assessment of medical trainee competence for independent clinical work. Acad Med. 2008;83(10 suppl):S89–S92.
4. Epstein RMAssessment in medical education. N Engl J Med. 2007;356:387–396.
5. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus SGeneral competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21:103–111.
6. Ericsson KAAcquisition and maintenance of medical expertise: A perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90:1471–1486.
7. Van der Vleuten CP, Sluijsmans D, Joosen-ten Brinke DMulder MCompetence assessment as learner support in education. In: Competence-based Vocational and Professional Education, Technical and Vocational Education and Training: Issues, Concerns and Prospects. 2017:Basel, Switzerland: Springer International Publishing; 607–630.
8. Konopasek L, Norcini J, Krupat EFocusing on the formative: Building an assessment system aimed at student growth and development. Acad Med. 2016;91:1492–1497.
9. Turner TL, Bhavaraju VL, Luciw-Dubas UA, et alValidity evidence from ratings of pediatric interns and sub-interns on a subset of pediatric milestones. Acad Med. 2017;92:809–819.
10. Lomis K, Amiel JM, Ryan MS, et alImplementing an entrustable professional activities framework in undergraduate medical education: Early lessons from the AAMC Core Entrustable Professional Activities for Entering Residency Pilot. Acad Med. 2017;92:765–770.
11. Brown DR, Warren JB, Hyderi A, et alFinding a path to entrustment in undergraduate medical education: A progress report from the AAMC Core Entrustable Professional Activities for Entering Residency Entrustment Concept Group. Acad Med. 2017;92:774–779.
12. ten Cate OEntrustment decisions: Bringing the patient into the assessment equation. Acad Med. 2017;92:736–738.
13. Sklar DP, Lee RCommentary: What if high-quality care drove medical education? A multiattribute approach. Acad Med. 2010;85:1401–1404.
14. Institute of Medicine. Crossing the Quality Chasm: A New Health System For the 21st Century. 2001.Washington, DC: National Academy Press;
15. Favreau MA, Tewksbury L, Lupi C, Cutrer WB, Jokela JA, Yarris LMConstructing a shared mental model for faculty development for the Core Entrustable Professional Activities for Entering Residency. Acad Med. 2017;92:759–764.
© 2017 by the Association of American Medical Colleges