Article In Brief
Neurology residency program directors discuss the impact the change toward a pass-fail score on Step 1 of the USMLE will have on criteria for selection into their programs.
Some neurology residency directors were taken by surprise last month (February) when they received notice from the United States Medical Licensing Examination (USMLE) that the Step 1 test was being changed from a graded score to “pass/fail,” starting January 1, 2022.
Licensing officials said the change was to stop residency programs from relying so heavily on the scores when determining whom to interview, but directors said they were concerned that they would have no objective measures in which to compare students, particularly since more medical schools are moving to a pass/fail grading system.
“I don't think anyone has a definitive answer on how we will choose residents if they don't have Step 1 as a potential screen to help identify applicants to interview,” said Ajitesh Ojha, MD, assistant professor of neurology and program director of the neurology program at the University of Pittsburgh Medical Center.
Dr. Ojha's program had 700 applicants for seven slots last year. “There is some emphasis on recommendation letters and personal statements, but those are pretty generic applicant-to-applicant. There may be more emphasis on research and service in programs, as well as extracurricular activities, but those opportunities are different from school to school.”
Step 1 is the first of a three-part series from the USMLE and is given during the second year of medical school. It is a multiple-choice test and scores range from 1-300 with a passing score of 194. The organization issued a statement announcing the change on February 12. Dave Johnson, the Federation of State Medical Boards chief assessment officer, said that the real push to change the test to pass/fail started last March after the Invitational Conference on USMLE Scoring (inCUS).
At the InCUS meeting, which included 60 members from the American Medical Association, the Educational Commission for Foreign Medical Graduates, and the two groups that oversee the USMLE—the Federation of State Medical Boards and the National Board of Medical Examiners. A report from the two-day gathering was released in June, suggesting that the USMLE consider changing how they reported on Step 1.
There was a public comment period, and the final decision was announced February 12, Johnson said in an interview monitored by a public relations person. He said they tried to be as transparent as possible about the process but that residency directors might not have expected the change to come so quickly. The response had been “largely positive,” he said.
“This creates an opportunity for the broader community to think about what kind of system they want for transitioning medical students into residency,” Johnson said, adding that the Step 2 Clinical Skills scores would remain in place. “Hopefully this triggers a system-wide conversation.”
The Medical School Curriculum
Residency directors, who were interviewed for this article—many of whom had not-so-fond memories of studying for the Step 1 test—thought it was a good move in terms of improving medical school education overall. There were concerns about the test being unfair to socioeconomically-challenged students as well as underrepresented minorities, who classically do not score as well.
Some residency directors said the test did not work well with the medical school curriculum. Blair Ford, MD, FAAN, professor of neurology and neurology residency director at Columbia University Medical Center, said too many medical schools were teaching to the test, and would build in weeks of board prep, which was disruptive.
“The test was not used in the way it was intended, to judge general skills—it was used as a screening tool for residency programs to assess and then invite applicants to interview,” he said. “The issue is: It doesn't predict which students would be a good physician.”
How to Judge Applicants
All the neurologists interviewed had concerns about how they could fairly judge residency applicants, particularly since many medical schools have turned to pass/fail grading and are no longer releasing student rankings. They worried that too much emphasis would be placed on where the student went to medical school, and that unconscious bias would affect choices without any objective measures to use as a guidepost.
Rafael H. Llinas, MD, FAAN, professor and chair of the department of neurology at Johns Hopkins Bayview Medical Center and director of the neurology residency program, said the test was stressful for students and some subspecialties used Step 1 to create a hard line, saying that they would not interview anyone with less than a specific score.
He said when he reviews neurology candidates, he uses the scores as a “little push up or down,” particularly when so many schools no longer have letter grants or release ranking scores.
“So you get a lot of applicants from Stanford or Yale and it's all pass, pass, pass, and they all have some generic dean's letter and then you take away an objective measure we could use,” he said.
The change may place too much of an emphasis on the reputation of the medical school, meaning residencies may miss out on “scrappy” students who stand out at lesser-known schools.
“Some may do research to separate themselves, but some students may have to do work/study or may have family obligations or other stressors. I worry for those people, we want the type of residents who cared enough to fight their way up. I worry that we are kicking the can backward.”
What About Unconscious Bias?
Dr. Llinas also worried that, without any objective scores from Step 1, unconscious bias may play more of a role in the interview process, hurting international students, women and minorities.
“Unconscious bias is a real thing, and you have to find a way to fight it,” he said. “One thing Step 1 did was help women get into neurosurgery and orthopedic surgery,” he said. “They weren't welcomed, but what are you going to do when you have these women with spectacular board scores? They started letting them in.”
Zachary N. London, MD, FAAN, clinical professor and program director of the neurology residency program at the University of Michigan Medical School, said they typically get between 400 and 600 applicants for nine slots. Some schools, like the University of Toledo, are giving students evaluations based on professionalism and interpersonal skills. He would like to see more of that, he said.
“This is going to benefit the big name schools,” he said. “But it will allow the schools to redesign the curriculum so that they are focused on learning clinical skills and the things medical doctors need to know, rather than preparing for a single high-stakes exam.”
He worried that moving to pass/fail would particularly hurt international students, who are already at a disadvantage because of the stigma of the language barrier, he said.
“We're going to end up weighing heavily on other things—are they an AOA [Alpha Omega Alpha, the medical honor society], which is usually the top 15 percent of the class? Research and publication and community service will count more, but I don't know if those are the things students want us to be emphasizing. You've got the letter of recommendation and their personal statements and their involvement in extracurriculars, those are the meat, those are the things that take the deep dive on my part, but I can't do a deep dive on 450 applicants,” said Dr. London.
“Plus, people who do research are usually from higher socioeconomic backgrounds and have time to spend on those sorts of things, and that's something we need to take into consideration.”
Matches are already set for this year, but medical schools and residency programs would need to figure out how they will adjust to the Step 1 change before it starts in 2022, directors said.
One impact might be that medical schools might start encouraging students to take the Step 2 Clinical Skills earlier so that the scores could be included in residency applications, doctors said.
Dr. Ojha, of UPMC, said it helps that neurology is not as competitive as neurosurgery or orthopedic surgery, which used the scores as a cut-off point. But he worried that they might change Step 2 to pass/fail in the future.
“The goal and the ideal is that this will encourage programs to push for a more comprehensive review of applicants,” he said. “But what does that mean? It's going to mean different things for different programs. It might be more beneficial for some programs to look at research, others to look at Step 2, or extracurricular programs.”
It might not be so bad to abolish Step 1 altogether in the future, said Dr. Ford, from Columbia. He gets 650 applicants for 10 slots.
“Step 1 is about test-taking skills, and not if they can collaborate and work in teams, can they work in a professional environment?” he said. “I'd like to see a more creative test looking at problem solving, ethics, professionalism. You could have questions that post actual problems—scenarios describing a situation, or questions based on critical reading – physicians have to assimilate and prioritize so much information in medicine, and those are the skills that are critical.”
Johnson, the assessment officer from FSMB, said the organizations will be monitoring how students, schools and residency programs react to the change. He said the Coalition for Physician Accountability is trying to pick up the mantle and create a more holistic approach to screening selection.
Dr. Ford said there are many obstacles in figuring out ways to judge students, but he's looking forward to the challenge.
“We're going to have to adjust, and we'll develop a new system, and it may not be perfect, but we'll share best practices and talk to each other and hopefully end up with a positive change,” he said.