Total knee arthroplasty (TKA) is a common orthopaedic procedure practiced both in community and academic settings. All orthopaedic surgery residents are expected to learn and become competent in this procedure, as demonstrated by the inclusion of the procedure in the Accreditation Council for Graduate Medical Education (ACGME) Milestone program as a major orthopaedic procedure.1 The ACGME Milestones were first published in 2013 as a concerted effort to streamline resident education and provide objective goals to evaluate surgical competency.1,2 To achieve level 1 competency, a resident should be able to perform simple tasks such as a basic history and physical examination, prescribe nonsurgical treatments, provide basic perioperative care, and list potential complications.1,2 A resident of level 5 experience (considered an expert in their field) should be able to perform two or more surgical approaches to the knee and complex primary and simple revision TKA procedures, be able to develop complex postoperative management plans, and be able to surgically treat complex complications including periprosthetic fractures.1,2 A requirement for graduation would be performance at level 4 competency. The Milestones, however, are designed to address the completion of an entire case rather than a step breakdown evaluation, and the broad categories and the lack of definable skills may make it difficult to place someone properly on the evaluation scale.
Trainee involvement in TKA has been the subject of numerous studies.3-6 However, there are no studies evaluating the most common technical learning obstacles for residents and fellows. Identifying these factors may benefit the future of TKA education for both faculty and trainees alike. Therefore, the aim of this study is to determine the most common technical obstacles encountered during TKA learning. We were interested in identifying particular steps that could be problematic to trainees to better focus teaching efforts. We hypothesize that evaluating procedures on a step-by-step breakdown will result in better identification of difficulties and ways to address them.
Methods
This is a prospective, cohort observational study performed within a single orthopaedic residency program from September 2017 to April 2018. The faculty, residents, and fellows were asked to participate in the survey and data collection. Informed consent was acquired before the initiation of data collection. An explanation was provided to the trainees about each step and how they were to be performed. After routine primary TKA, the faculty (n = 3) who participated in the case completed an assessment survey of the resident/fellow in the case. The questionnaires took less than 5 minutes to complete with performance scored on a 0 to 5 scale based on performance proficiency ranging from 0 (not competent/not permitted to perform) to 5 (technique mastered) (Figure 1). The surveys were kept within the operating rooms, and the completion of the survey was performed immediately after the completion of the case, to prevent any recall bias. After the completion, the surveys were deposited within a collection box with the inability to modify responses once entered. Evaluations were performed without feedback or notification to the trainee to preserve survey accuracy. The 10 steps evaluated were as follows: distal femoral cut, tibial alignment guide placement, tibial cut, femoral sizing and rotation, use of a saw, retractor placement, soft-tissue technique, patellar resurfacing, tibial tray preparation, and femoral implant placement. These were subjectively selected by the participating attendings as the most critical portions of the case.
;)
Figure 1: Diagram showing the survey provided to an attending after the completion of a case with resident involvement. If a portion was not performed, the question was left blank and marked as N/A (given a score of 0/5 for analysis purposes). TKA = total knee arthroplasty
Inclusion criteria included any primary TKA during which a resident or fellow actively participated. Exclusion criteria included revision TKA, complex primary total knee cases not appropriate for resident/fellow primary assistance with the procedure, and lack of resident/fellow coverage for the entirety of the case. A complex primary was defined as those with previous surgical interventions to the knee and those with severe deformities (>15° varus/valgus, >15° flexion contracture). In addition, any partially completed forms or forms completed in a delayed fashion were eliminated from evaluation. After application of all inclusion and exclusion criteria, individual performance on 10 major technical steps in primary TKA was evaluated by fellowship-trained attending surgeons on a standardized, deidentified survey.
Study participants were divided into two comparison groups based on year in training multiplied by the number of cases performed: group 1 (junior—n = 44) was <20, whereas group 2 (senior—n = 59) was >20. The differentiation between junior (<20) and senior (>20) trainees was established after evaluating the entire resident class and determining that the median experience level was 20 (years of training × cases seen). This is why this value was chosen as the cutoff between senior and junior. Mean scores were calculated for each technical step and for the total (sum of scores) for each group. Cuts were evaluated for how even and level they were using an extension block guide after the cut. Any rocking, gapping, or unevenness was deemed to be an “uneven cut.” In addition to this, the amount of cutting guide motion, saw binding, and the need to start and stop the saw would all contribute to their scoring. The scores based on whether the cut was uneven were based on subjective evaluation after observing the mentioned features and were not quantified by an amount. The completion of the case was defined as being able to perform all steps of a case without the assistance of an attending physician.
Data were analyzed using a 2 (group) × 10 (question) multivariate generalized linear model for nonparametric data. Notable effects of the group and question were followed up by subsequent univariate analysis between the groups for each question and within the groups among the 10 questions. Final univariate pairwise comparisons were analyzed using a Bonferroni post hoc adjustment. In addition, we performed correlation coefficients for all questions, comparing scored techniques, year in training, cases observed, and total experience (years in training × cases observed). To determine the relationship between how trainees scored on each technique and the relationship between technique performance and experience, a Spearman rho correlation for nonparametric data was used. Correlations strength was defined as weak (r = 0 to 0.3), moderate (0.3 to 0.7), strong (0.7 to 0.8), and very strong (>0.8). Type I error was set at alpha = 0.05 for all analyses.
Results
A total of 103 surveys were filled out for the evaluation of fellows and residents by three faculty members. A total of 44 surveys were evaluated for the junior resident group, and 59 surveys were evaluated for the senior resident group. When comparing the surgical technique of the junior residents with that of the senior residents, our analysis revealed a significant effect of both experience level and technical step (P < 0.05). Subsequent analysis revealed a significant difference between the experience groups for all questions with the senior experience group scoring higher (P < 0.05). The comparison of scoring for each question within the respective groups saw Q1 (distal femoral cut), Q3 (tibial cut), and Q5 (use of saw) scoring the highest for the junior experience group and Q1 (distal femoral cut), Q4 (femur sizing & rotation), Q7 (soft-tissue technique), Q9 (tibial tray preparation), and Q10 (femoral implant placement) scoring the highest for the senior group (P < 0.05) (Figure 2).
Figure 2: Chart showing the summary of the proficiency scores assigned to junior and senior residents for each aspect of the case. Data are presented as means ± SD for proficiency scores for junior (cases seen × year in training < 20) and senior (cases seen × year in training > 20) experience groups. *Pairwise differences BETWEEN experience groups were detected for all questions (P < 0.05). Letters over bars indicate differences in score between questions WITHIN each group with “like” letters indicating no significant difference from question to question (P < 0.05).
Overall, the highest scores for both junior and senior experience levels were achieved in performing the distal femoral cut and tibial cut. The lowest scores for both junior and senior experience levels were achieved in performing placement of the tibial alignment guide, placing retractors, and patellar resurfacing. The largest difference from junior to senior levels was made in performing sizing of the femur and setting rotation, tibial tray preparation, and femoral implant placement. The least difference was found in use of the saw and soft-tissue technique (Table 1).
Table 1: Mean Scores of Junior & Senior Trainees
We also wanted to evaluate the flow of a case by determining whether training or year of experience had any correlation with an overall technical mastery. The results demonstrated that skills competency and technique were not only related to year of training and experience but also to each other independent of experience. If residents were identified to struggle with one aspect of the surgery, they were statistically more likely to struggle with all portions of the case. When evaluating the relationships between the steps, the scores on every step were linked to the previous and following step in all cases and all experience levels (P < 0.05). Some aspects, such as the sizing of the femur and setting of the rotation, even dictated the success of the rest of the case with high statistical significance (P < 0.01).
Discussion
Our results demonstrate that year of training correlates with competency in tasks related to performing a TKA, but most residents, even those in senior positions, may not accomplish competency of the entire procedure. All residents were evaluated from September to April of their assigned experience level, defined as the midpoint in training. Senior residents perform all portions of a case better than junior residents, but not necessarily to the level that is deemed mastered (level 4). In addition, all steps were closely correlated with each other; if residents struggled with one step, they likely struggled with all of them. Finally, we have demonstrated that residents have the most difficulty with conceptual tasks, rather than technical ones.
In evaluating the steps, we broke down tasks based on whether it was a technical skill or a conceptual skill because some think that the key to a successful operation is up to 75% decision-making.7 A technical skill is any psychomotor action acquired through practice and learning pertaining to a particular craft or profession.8,9 A nontechnical skill (in this case deemed a “conceptual skill”) is defined as a collective term used to describe the skills and behaviors encompassing situational awareness, decision-making, communication, teamwork, and leadership.8 Technical skills include the distal femoral cut, tibial cut, use of the saw, retractor placement, and soft-tissue technique. Conceptual skills include placement of the tibial alignment guide, tibial tray preparation (including sizing and rotation), patellar resurfacing (including sizing, positioning, and rotation), femoral sizing and setting of rotation, and femoral implant placement. Of note, two faculty members used a free-hand technique for patellar resurfacing and one used a guided technique. For the evaluation of technical skills, the standard for tibial alignment included neutral overall limb alignment and an anatomic posterior slope, and external rotation allowing the middle of the tibial tray to align with the middle of the patellar tendon. For the femur, the benchmark of alignment was 5° external rotation with neutral overall alignment of the limb. Finally, regarding techniques performed during the tibial cut, exposing the saw blade too aggressively for the cut into surrounding soft tissues (ie, the medial collateral ligament (MCL) [too far medial], posterior capsule [too deep], or patellar tendon [too anterolateral]).
Considering this, our results show the largest strides made in conceptual tasks (tibial tray preparation, femoral guide placement, and femoral implant placement). It also shows the most difficult tasks encountered at both levels were conceptual tasks (patellar resurfacing, tibial alignment guide placement, retractor placement). Retractor placement could be considered both a technical and conceptual task depending on the interpretation. Teaching has traditionally focused on technical skills,10 but this implies conceptual tasks may require more teaching focus but also may vary between attendings.
In the authors' opinion, one of the most important technical tasks is the use of the saw. When evaluating trainees on their saw technique, items that were taken into account included the smoothness of the cut, the amount of binding the saw experienced within slotted cutting guides, and the need for repeat cuts to accomplish an evenly cut surface. Because of these very defined criteria, the progress made in training year was the least with the saw technique. This demonstrates that experienced arthroplasty surgeons think trainees have not yet achieved proficiency with a power saw similar to themselves. The literature has shown saws that get bound up in constrained cutting guides can result in movement of the cutting block by up to 1 mm.11 In addition, saw toggle and a lack of saw control can result in cutting errors as large as 0.8 mm, which may result in major cutting errors and the potential for an unbalanced knee.11
The ACGME Milestones were first published in 2013 to provide objective goals to evaluate residents' competency.1 These were further modified and released in 2015 in a joint effort from the ACGME and American Board of Orthopaedic Surgeons.2 The Milestone Project stated that “The Milestones provide a framework for the assessment of the development of the resident physician in key dimensions of the elements of physician competency in a specialty or subspecialty,” but they cautioned that the Milestones “neither represent the entirety of the dimensions of the six domains of physician competency, nor are they designed to be relevant in any other context.”2 That being said, some have argued that an added benefit is that residents themselves are able to use the Milestones as a way to evaluate their own learning and make adjustments as necessary.12 However, controversy still exists as to whether there is any validity to these assessments and whether they are truly able to assess if a trainee surgeon is competent, given the broad categories evaluated.13 By comparing the Milestones with our study, we were able to demonstrate that looking at the completion of a procedure as a whole task may minimize the progression within a specific procedure. As expected, we found that residents who were in a junior year of training were less proficient with all portions of the case when compared with senior residents. However, we also found that despite the perception that senior residents should have mastered a simple primary TKA earlier in training, when broken down on a technical step-by-step basis, this was not being accomplished. By considering this, we think that progress is better evaluated in this more detailed step-by-step basis. We think this is a better way to identify where residents may be stalling to make progression to completing a full procedure competently.
Multiple studies have shown that increased case volume improves the outcomes and efficiency and minimizes complications.14-17 Therefore, although the results of the comparison between the resident levels show senior residents are able to perform all portions of a case more competently than junior residents, more valuable information can be acquired through the steps breakdown. As a senior-level resident, issues were most often encountered during tibial alignment guide placement (Q2), use of the saw (Q5), retractor placement (Q6), and patellar resurfacing (Q8). In all of these categories, most senior residents were deemed to have not mastered all of the techniques. This is in direct contrast to evaluations of the Milestones, where most senior-level residents thought they had achieved a level 4 mastery, which corresponds to not only mastering a primary total knee but also simple revision knees.17,18 We have demonstrated that in isolation, this mastery may not be true and could serve as a deficiency when these residents enter independent practice. By addressing deficiencies on a step-by-step breakdown, you give residents a more focused skill to work on intraoperatively and you ensure that all portions of the case can truly be performed independently.
Finally, in evaluating a link between the steps, we identified that if residents were to perform poorly with one portion of the case that we evaluated, they were more likely to score poorly with all portions of the case regardless of their experience. We think this is valuable information to acquire when training residents because it may allow early identification of those who are not performing adequately. Studies have previously questioned the accuracy between case logs and attending evaluation of performance,19 but other studies have good validity between self-evaluation in comparison with attending evaluations.20 This study takes this one step further by being able to identify residents having difficulty in early steps of the case to allow for early correction, an area of research currently lacking in the orthopaedic literature. Patients who underwent TKA surgery at a teaching hospital, under resident assistance, were more likely to be in the operating room for longer and have higher perioperative resource consumption.6 Therefore, identifying and improving residents' performance is not only important from a teaching perspective but also from a patient's outcome perspective.
Limitations of the study include a lack of instrument validation, and the 10 steps listed for grading may not encompass the entirety of the case and have not been formally validated before the study. The use of the scoring scale, however, was similar to previous investigations using Likert-based scoring for evaluation of technical proficiency in clinical and nonclinical settings.21-23 This has been a standardized validated practice in clinical research for collecting qualitative data in several current and past investigations. In addition, we added the score of 0 for those not able, or permitted, to complete a particular technique. In addition, the interobserver variability may account for some differences in the score. Some attending surgeons are more likely to scrub with senior versus junior residents, and therefore, the grading style may not be consistent across the population. Interpersonal relationships and trust may affect the scores; however, attempts to limit this were made by making the surveys measure technical aspects only. Expectedly, not applicable (N/A) responses for the junior surveys were much higher than those for the senior surveys, indicating that junior residents were not allowed to perform certain portions of a case. Had they been allowed to perform some of the more “technically demanding” portions of the case, the findings might have differed somewhat. It is important to note, however, that the residents were not penalized for their previous step performance. Instead, if the supervising surgeon thought it would be unsafe to the patient or the resident did not possess the skill level to perform the task (based on year of training, and in general not on individual competence), then a score of 0 would be awarded. Goals of future studies are to validate the proposed evaluation scale and formalize more objective ways of evaluating surgical skills (ie, using video review, real-time third-party evaluators).
Conclusion
We have shown that residents who were in a junior year of training were less proficient with all portions of the case when compared with the senior residents. We have also shown that despite historical evaluations to the contrary, most senior-level residents cannot necessarily perform all steps of a TKA proficiently, potentially setting them up for issues in independent practice. This evaluation tool can be used to help senior residents continue to make progress and ensure true proficiency and should be used in conjunction with the ACGME Milestones.
References
1. Stern PJ, Albanese S, Bostrom M, et al.: Orthopaedic surgery milestones. J Graduate Med Educ 2013;5(1 suppl 1):36-58.
2. Stern PJ, Bostrom M, Day CS, et al.: The Orthopaedic Surgery Milestone Project. The Accreditation Council for Graduate Medical Education and The American Board of Orthopaedic Surgery. 2015.
http://www.acgme.org/portals/0/pdfs/milestones/orthopaedicsurgerymilestones.pdf. Accessed August 21, 2018.
3. Dwyer T, Wadey V, Archibald D, et al.: Cognitive and psychomotor entrustable professional activities: Can simulators help assess competency in trainees? Clin Orthop Relat Res 2016;474:926-934.
4. Haughom BD, Schairer WW, Hellman MD, et al.: Does resident involvement impact post-operative complications following primary total knee arthroplasty? An analysis of 24,529 cases. J Arthroplasty 2014;29:1468-72.e2.
5. Mayden CA, Anglin C, Kopp GD, et al.: Computer-assisted surgery simulations and directed practice of total knee arthroplasty: Educational benefits to the trainee. Comput Aided Surg 2012;17:113-127.
6. Lavernia CJ, Sierra RJ, Hernandez RA: The cost of teaching total knee arthroplasty surgery to orthopaedic surgery residents. Clin Orthop Relat Res 2000:99-107.
7. Spencer FC: Teaching and measuring surgical techniques: The technical evaluation of competence. Bull Am Coll Surg 1978;63:9-12.
8. Agha RA, Fowler AJ, Sevdalis N: The role of non-technical skills in surgery. Ann Med Surg (Lond) 2015;4:422-427.
9. Trumble WR, Stevenson A, eds: The Shorter Oxford Dictionary, ed 5, Oxford, England: Oxford University Press, 2002.
10. Yule S, Flin R, Paterson-Brown S, Maran N: Non-technical skills for surgeons in the operating room: A review of the literature. Surgery 2006;139:140-149.
11. Otani T, Whiteside LA, White SE: Cutting errors in preparation of femoral components in total knee arthroplasty. J Arthroplasty 1993;8:503-510.
12. Tosti R: Will the new milestone requirements improve residency training? Am J Orthop 2013;42:E109-E110.
13. Van Heest AE, Dougherty PJ: CORR curriculum: Orthopaedic education: Operative assessment and the ACGME milestones: Time for change. Clin Orthop Relat Res 2015;473:775-778.
14. Cheng CH, Cheng YT, Chen JS: A learning curve of total knee arthroplasty (TKA) based on surgical volume analysis. Arch Gerontol Geriatr 2011;53:e5-e9.
15. Critchley RJ, Baker PN, Deehan DJ: Does surgical volume affect outcome after primary and revision knee arthroplasty? A systematic review of the literature. Knee 2012;19:513-518.
16. Schroer WC, Calvert GT, Diesfeld PJ, Reedy ME, LeMarr AR: Effects of increased surgical volume on total knee arthroplasty complications. J Arthroplasty 2008;23:S61-S67.
17. Jeray KJ, Frick SL: A survey of resident perspectives on surgical case minimums and the impact on milestones, graduation, credentialing, and preparation for practice. J Bone Joint Surg Am 2014;96:e195.
18. Bradley KE, Andolsek KM: A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation. Int J Med Educ 2016;7:11-18.
19. Falchikov N, Boud D: Student self-assessment in higher education: A meta-analysis. Rev Educ Res 1989;59:395-430.
20. Trajkovski T, Veillette C, Backstein D, Wadey VMR, Kraemer B: Resident self-assessment of operative experience in primary total knee and total hip arthroplasty: Is it accurate? Can J Surg 2012;55:S153-S157.
21. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 2018;89:8-18.
22. Spooren P, Mortelmans D, Denekens J. Student evaluation of teaching quality in higher education: Development of an instrument based on 10 Likert-scales. Assess Eval Higher Educ 2007;32:667-679.
23. Hodgins JL, Veillette C: Arthroscopic proficiency: Methods in evaluating competency. BMC Med Educ 2013;13:61.
