I was a new program director in 2001 trying to figure out what I was doing. There are endless moving parts in a residency, but I was particularly interested in how to conduct resident evaluations. Program directors are required to meet with each resident twice a year to provide a formal assessment of how each is doing. It is a lot of work, but it is the best opportunity to mentor the people who have entrusted me with their professional future.
There were essentially no guidelines on how to evaluate residents before that year. The Accreditation Council for Graduate Medical Education (ACGME) was created two decades earlier, but it more or less left it up to the program directors to act like Romans judging gladiators. It was thumbs up or thumbs down. Of course, this was a setup for abuse, and some program directors in other specialties were legendary for holding back residents from graduation until they felt like they were ready to practice independently.
At the start of the new millennium, however, ACGME required program directors from all specialties to evaluate each resident on six core competencies, including practice-based learning, systems-based practice, and professionalism. Needless to say, there were exactly zero metrics for how to judge these skills. I remember the chairman of the board of directors for ACGME making a comment that the whole process had taken off while everyone was still reading the manual on how to fly the plane.
After a few years of complaining and pontificating, ACGME decided that a one-size-fits-all solution for evaluating the competency of residents from all specialties left a lot to be desired. So they asked the academic leaders from each specialty to create a committee for developing a specialty-specific competency evaluation, which became known as the Milestones Project. Subsequently, the committee for emergency medicine came up with 23 skills for assessment in 2013, all of which had to be scored and recorded for each resident on the ACGME Data Accreditation System. Instead of scoring the relatively meager six skills before, program directors had to evaluate each resident on pharmacology, ultrasound, disposition, vascular access, diagnosis, diagnostic studies, team management, etc.
Talk about being careful what you wish for. Imagine you are a program director with a three-year curriculum that has 13 residents in each class. That means you will make 897 clicks on your computer to enter milestone data on the ACGME website each fall. You get to do it all over again in the spring. That means each year you get to make 1794 clicks for ACGME. But wait, there's more.
Clicks and More Clicks
Not to be outdone by the ACGME, your hospital also wants you to record these data for its internal database. You have to go on its website and do another 1794 clicks for a grand total of 3,588 per year. But is all this clicking actually producing better doctors? Quite frankly, I doubt it.
In the age of big data, saying that collecting more data is not the right thing to do can make administrative wonks throw you to the wolves. They live to create more and more data that they can analyze and put into reports that no one will read. I have created thousands and thousands of pages of resident evaluations that only spent a few minutes in the light of day and then were dumped onto a virtual hard drive. Some of the stuff on these evaluations is obviously important. It is the best tool I have to make sure residents are getting enough opportunities for procedures and completing their nonclinical duties. But none of the residents reads through the results of all of the milestone evaluations. They just look at the overall average and how it compares with the rest of the class. In the end, they want a single number. Or if you think about it, they want the thumbs up or thumbs down.
What I do know from these evaluations with residents over all these years is that what really moves the needle for performance are the comments from faculty members and peers that accompany all the clicks we do for the milestones. Three or four comments about how the resident needs to pick up more charts, show up on time, be kind to nurses and patients, or not get jaded will get anyone's attention. A million data points will never have the same impact as a few choice words.
Share this article on Twitter and Facebook.
Access the links in EMN by reading this on our website, www.EM-News.com.
Comments? Write to us at firstname.lastname@example.org.
Dr. Cookis the program director of the emergency medicine residency at Palmetto Health Richland in Columbia, SC. He is also the founder of 3rd Rock Ultrasound (http://emergencyultrasound.com). Friend him atwww.facebook.com/3rdRockUltrasound, follow him on Twitter@3rdRockUS, and read his past columns athttp://bit.ly/EMN-Match.