Journal Logo

AOA Critical Issues in Education

The Content and Accessibility of Orthopaedic Residency Program Websites

Sherman, Nathan C. MD, MBA1,2; C. Sorenson, Jacob BS1,2,a; M. Khwaja, Ansab MD1,2; L. DeSilva, Gregory MD, FAOA1,2

Author Information
doi: 10.2106/JBJS.OA.20.00087
  • Open
  • Disclosures
  • SDC

Abstract

Introduction

Orthopaedic surgery continues to be one of the most competitive specialties for graduating medical students with an annual match rate of approximately 75% to 80%. The match rate has remained stable despite yearly increases in both the number of applicants and number of applications1. The match is designed to provide applicants and programs with an unbiased selection based on their relative rank lists, and significant effort goes into the formulation of these rank lists before match day and even before the interview season.

During the application process and residency search, prospective applicants have nominal resources to evaluate programs, often relying on mentors or current residents with outside knowledge of programs. For prospective applicants and residency programs alike, the primary objective of the interview process is to evaluate candidacy while serving as an opportunity for formal introduction and information gathering. Before the interview, applicants will often turn to online resources in order to learn more about the programs to which they are applying. The Electronic Residency Application Service (ERAS) is the universal application processing service that prospective orthopaedic residents will review, and the ERAS website is a valuable resource with links to orthopaedic residency programs and information about the application requirements of each program accredited by the Accreditation Council for Graduate Medical Education. Another resource is the Fellowship and Residency Electronic Interactive Database (FREIDA) available through the American Medical Association for a fee, but sometimes available without cost through select institutions. Many applicants, however, will supplement their research into residency programs with an online search engine such as Google.

For the 2020 residency application cycle, 1,265 United States and Canadian Medical Graduates (USGs) applied to orthopaedic surgery residency programs with each applicant applying to a mean of 81.5 programs2. Each program then reviewed an average of 548.5 USG applications and an additional 39.0 applications from International Medical Graduates (IMGs)2. Eight hundred forty-nine orthopaedic surgery residency positions were offered, and of these positions, all but 5 were filled by 714 MD seniors and graduates, 118 DO seniors and graduates, and 12 IMGs3.

Although studies of the content and accessibility of nonorthopaedic residency websites have been performed4-7, to the authors’ knowledge, no study has specifically evaluated the content and accessibility of orthopaedic surgery residency websites since 20018. Orthopaedic surgery fellowship websites, however, have been analyzed9-23, and this study uses a similar protocol9,10,14,23 to assess orthopaedic residency website content and accessibility.

The primary objective was to measure the content and accessibility of recruitment and educational domains and compare the results with previous data. A secondary objective was to compare applicable domains with orthopaedic fellowship websites. It was hypothesized that the website content and accessibility of orthopaedic surgery residency programs would be greater than the lone previously published article on the subject and comparable with that observed in orthopaedic fellowship programs.

Methods

Existing allopathic and osteopathic orthopaedic surgery residency programs in the United States were identified on the ERAS website. Absent, direct, and indirect links or links requiring more than two clicks were codified. All data were recorded between the dates of March 25, 2020, and April 2, 2020, and data analysis was performed on Microsoft Excel, version 16.35 (Appendix 1, http://links.lww.com/JBJSOA/A202).

Using the criteria adapted from previous studies on fellowship program websites, orthopaedic residency program websites were evaluated based on two domains: recruitment and educational content9,10,14,23. Each domain contained 10 criteria, selected for their potential to influence recruitment and education, respectively. Recruitment domains included (1) application requirements, (2) current fellows, (3) current residents, (4) faculty listing, (5) location description, (6) past residents and their rmployment, (7) program contact information, (8) program description, (9) salary and benefits, and (10) selection criteria. Educational domains included (1) call schedules, (2) didactics schedule, (3) evaluation criteria and competencies, (4) examples of research, (5) journal club, (6) meetings and courses, (7) office and clinic descriptions, (8) operative experience and/or case log, (9) research requirements or opportunities, and (10) rotations and curriculum. Data were coded as “0” for absent or “1” for present based on the presence of these criteria in each program’s website. Quantitative or qualitative descriptions of these criteria were sufficient to warrant a score of “1,” with the exception of operative experience. Unless a case log or the number of cases was present, operative experience would receive a score of “0” as qualitative descriptions of operative experience are insufficient to receive a score of “1.”

On the interview trail, the presence of fellows is a talking point that is commonly addressed. Although the merits and detriments of fellow presence on a resident’s educational experience may be argued, it is an important dynamic which may affect prospective applicants and was thus coded as a variable under the recruitment domain. The presence of fellows was categorically assessed, and the websites were given a “0” if no information on fellows was provided or a “1” if there was information available. Programs without fellowships were required to mention the absence of fellows to receive a “1” in this category.

No funding was sought or obtained to support this investigation. The departmental Institutional Review Board declared this study exempt from oversight because the data obtained for this study were freely accessible and publicly available.

Results

One hundred eighty-nine orthopaedic surgery residency programs were accredited and available for application through ERAS. Only six programs (3.2%) did not have a website identifiable through ERAS or Google searches, leaving a final sample size of 183 websites (Table I). Two programs were not participating in the match for 2020, and at the time of data collection, 11 programs had updated their listings to reflect that they were no longer accepting applications for this cycle, inactivating the hyperlink to their program’s website.

TABLE I - Online Residency Program Accessibility
Link Type Electronic Residency Application Service % Google %
Absent/nonfunctioning link 21 11 6 3
Website with direct link 145 77 181 96
Website with indirect link 23 12 2 1

Approximately 73.3% of all recruitment data were present on the 183 accessible websites. The most commonly identified recruitment data on orthopaedic surgery residency websites was program contact information, program description, and application requirements, with 98.9%, 96.2%, and 93.4%, respectively. Salary and benefit information was present in 74.3% of websites. Lists of faculty were present in 90.2%, residents in 92.9%, and fellows in 39.3% of websites. Descriptions of selection criteria were scarcely observed (8.7%), but past residents and their employment were more readily available (63.4%) (Fig. 1).

Fig. 1
Fig. 1:
Availability of recruitment content.

Only 44.9% of all education data were available. Schedules for call, didactics, and journal club were present in 43 (23.5%), 121 (66.1%), and 103 (56.3%) websites, respectively. One hundred twenty-four programs identified meetings or courses that residents attended (67.8%). The office and clinical schedule as well as a quantitative representation of operative experience or case log was present in 42.1% and 11.5% of program websites, respectively. One hundred fifty-nine programs mentioned rotations/curriculum (86.9%), but only 13 reported the evaluation criteria or competencies (7.1%). One hundred twenty-six programs stated research requirements (68.9%), but only 35 provided examples of research (19.1%) (Fig. 2).

Fig. 2
Fig. 2:
Availability of education content.

Only five recruitment domains and six educational domains were consistent between the original8 and current study. Of these domains, there was a statistically significant improvement in all criteria listed (p < 0.05) (Figure 3 and 4). Our data were compared with existing data for foot and ankle, hand, joint reconstruction, pediatric, shoulder and elbow, sports, and trauma orthopaedic fellowship websites (Tables II and III).

Fig. 3
Fig. 3:
2001 to 2020 comparison: recruitment content.
Fig. 4
Fig. 4:
2001 to 2020 comparison: education content.
TABLE II - Residency and Fellowship Website Comparison: Recruitment Content*
Recruitment Criteria Residency F & A Hand Joints§ Peds# S & E** Sports†† Trauma‡‡ Average SD
Application requirements 93.4% 47.9% 97.0% 82.8% 74.0% 77.8% 70.0% 41.0% 73.0% 19.9%
Current fellows 39.3% 14.6% 32.0% 25.9% 23.0% 27.8% 36.0% 35.0% 29.2% 8.1%
Faculty listing 90.2% 100% 64.0% 65.5% n/a 77.8% 93.0% 29.0% 74.2% 24.2%
Location description 75.4% 8.3% n/a n/a n/a n/a n/a n/a 41.9% 47.4%
Past residents/employment 63.4% 16.7% n/a 13.8% n/a 22.2% n/a 31.0% 29.4% 20.1%
Program contact info 98.9% 100% 85.0% 56.9% 97.0% 38.9% n/a n/a 79.5% 25.7%
Program description 96.2% 97.9% 100% 98.3% 100% 100% 100% 100% 99.0% 1.4%
Salary/benefits 74.3% 100% 28.0% 34.5% 31.0% 27.8% 30.0% 25.0% 43.8% 27.7%
Selection criteria 8.7% n/a n/a n/a n/a n/a n/a 16.0% 12.4% 5.1%
Average 71.1% 60.7% 67.7% 53.9% 65.0% 53.2% 65.8% 39.6% 59.6% 10.3%
*n/a = not available; Current Residents were removed from consideration in this figure.
F & A = Foot and Ankle: Khwaja et al., 2020;
Hinds et al., 2016;
§Young et al., 2018;
#Pediatrics: Davidson et al., 2014;
**S & E = Shoulder and Elbow: Young et al., 2016;
††Yayac et al., 2017;
‡‡Shaath et al., 2018.

TABLE III - Residency and Fellowship Website Comparison: Education Content*
Education Criteria Residency F & A Hand Joints§ Peds# S & E** Sports†† Trauma‡‡ Average SD
Call 23.5% 18.8% 32.0% 13.8% 34.0% 22.2% 19.0% 20.0% 22.9% 6.9%
Didactics 66.1% 45.8% 68.0% n/a 69.0% n/a 69.0% 53.0% 61.8% 9.9%
Evaluation criteria/competencies 7.1% 10.4% n/a n/a n/a n/a n/a 20.0% 12.5% 6.7%
Examples of research 19.1% 2.1% n/a 13.8% n/a 11.1% 20.0% n/a 13.2% 7.2%
Journal club 56.3% 41.7% 54.0% 31.0% 54.0% 44.4% 56.0% 47.0% 48.0% 8.8%
Meetings/courses 67.8% 18.8% 21.0% 67.2% 17.0% 66.7% 42.0% 39.0% 42.4% 22.4%
Office/clinic 42.1% 14.6% 38.0% 25.9% 57.0% n/a 30.0% 33.0% 34.4% 13.4%
Operative experience/case log 11.5% 16.7% 49.0% 96.6% 60.0% 88.9% 64.0% 76.0% 57.8% 31.1%
Research 68.9% 91.7% 75.0% 89.7% 83.0% 100% 61.0% 49.0% 77.3% 17.1%
Rotations/curriculum 86.9% 89.6% 28.0% 31.0% 63.0% 38.9% 28.0% 65.0% 53.8% 25.8%
Average 44.9% 35.0% 45.6% 46.1% 54.6% 53.2% 43.2% 44.7% 45.9% 6.1%
*n/a = not available.
F & A = Foot and Ankle: Khwaja et al., 2020;
Hinds et al., 2016;
§Young et al., 2018;
#Pediatrics: Davidson et al., 2014;
**S & E = Shoulder and Elbow: Young et al., 2016;
††Yayac et al., 2017;
‡‡Shaath et al., 2018.

Under recruitment content, orthopaedic residency websites listed salary/benefits more than all specialties other than foot and ankle fellowship websites. Residency websites listed past residents/employment more than fellowship websites but provided similar reports of application requirements, program contact information, and program descriptions. Orthopaedic residency programs presented data of current fellows in 39.3% of websites, the highest percentage observed in the studies considered, and more than one standard deviation above the mean.

Under educational domains, 65% to 68% of residency, joints and shoulder and elbow fellowship websites presented meetings/course information, which was more than one standard deviation above the mean. Residency websites trailed only foot and ankle fellowship websites in presentation of rotations/curriculum content with 86.9% of data reported compared with 89.6%. Residency websites consistently fell within one standard deviation of the mean for presentation of content related to office/clinic, evaluation criteria/competencies, research, examples of research, journal club, didactics, and call. Orthopaedic surgery residency programs, however, were more than one standard deviation below the mean regarding data of operative experience/case logs (11.5%).

Discussion

This is the first study in nearly 20 years to evaluate the content and accessibility of orthopaedic surgery residency program websites. In 2001, Rozental et al. used FREIDA, Yahoo (www.yahoo.com), AltaVista (www.altavista.com), and Excite (www.excite.com) to guide their search. The authors came across 154 accredited programs at the time and only 113 websites (73.4%)8. Since then, an additional 35 programs have earned accreditation, and all but six have accessible websites (96.8%). As hypothesized, the accessibility and availability of recruitment and educational content regarding individual orthopaedic surgery residency programs has significantly increased.

The use of internet has grown exponentially since 2001; therefore, this study must also be considered in the context of its era and can be treated as a standalone snapshot of the 2020 content of orthopaedic surgery residency websites. Approximately 73.3% of the recruitment domain and 44.9% of the educational domain content were easily accessible through internet search. This study provides a starting point for further refinement of content on individual websites, although only the presence of these domains and not the quality of the presentation was evaluated.

There is significant variability in the content presentation of orthopaedic residency and orthopaedic fellowship websites. Compared with orthopaedic fellowship websites, orthopaedic residency programs provide the most recruitment content but fall below average in presentation of education content. One contributor to poor educational content reporting is that residency programs report operative experience/case log at the lowest rate of any of the studies evaluated. However, this finding may be due to author interpretation because this study did not accept qualitative descriptions of operative experience as sufficient to warrant the criteria as present.

With the rapid proliferation and growing importance of internet resources, it is not surprising that orthopaedic surgery residency program websites have improved in the last 20 years. Perhaps what is more surprising is that not all programs use this opportunity to capitalize on brand marketing to attract applicants who may be interested in the program. As applicants may struggle to make themselves stand out in a crowded pool of qualified students and graduates in search of the same job, so too may programs have difficulty differentiating themselves from one another.

As the number of applications that programs review increases, the proportion of applications receiving in-depth review decreases24. To the authors’ knowledge, there is no research to suggest that better information from a website will limit the number of applications or applicants to orthopaedic surgery residency programs. However, uniform transparency through standardization of orthopaedic surgery residency program websites has the potential to benefit both prospective applicants and programs alike. Applicant access to more program content provides for better-informed decision-making. Furthermore, programs have the opportunity to use online resources to attract applicants and enhance their online presence.

Our study demonstrates that there is still opportunity for improvement in the online presentation of both education and recruitment content by residency program websites. To our knowledge, there are currently no transparency or standardization mandates for orthopaedic residency program websites. In the interest of improved mobile access and information-sharing, the authors recommend that influential orthopaedic residency education bodies, such as the American Orthopaedic Association’s Council of Orthopaedic Residency Directors, consider promotion of orthopaedic residency program website standardization.

There are several limitations to this study. All data were collected in late March to early April 2020, at which time approximately 11 orthopaedic surgery residency programs had already updated their ERAS listings to reflect that they were no longer accepting applications. Two were not accepting applications for the 2020 application cycle. Although data were collected during a short interval, we cannot confirm that websites were not altered by individual programs during the time of data collection. In addition, we used the criteria adapted from previous studies9,10,14,23 to perform this analysis, and there may be variation in author interpretation of the results. We acknowledge that “recruitment” and “education” are arbitrary categories and are not entirely as binary as implied. Rather, they are a categorical necessity to compare our work with that of others. Furthermore, the quality of the content presented by orthopaedic surgery residency programs was not directly evaluated, but the presence of content was. Therefore, although this study may allow direct comparison of program websites solely on the presence of content, the methodology does not support comparisons of website or program quality. A further limitation of this study is that no assessment of data accuracy or interrater reliability is provided because data were collected by a single member of our team during a short interval.

Conclusion

This is the first study since 2001 to assess the content and accessibility of orthopaedic surgery websites and demonstrate improvement in the content and accessibility from previous data. Compared with orthopaedic fellowship websites, orthopaedic residency programs provide the most recruitment content but fall below average in presentation of education content, particularly regarding reporting of operative experience. There continues to be opportunity for improvement in orthopaedic residency website content and accessibility from which both applicant and program may benefit.

References

1. Trikha R, Keswani A, Ishmael CR, Greig D, Kelley BV, Bernthal NM. Current trends in orthopaedic surgery residency applications and match rates. J Bone Joint Surg Am. 2020;102:e24.
2. Electronic Residency Application Service: Residency - Preliminary Data (ERAS 2020). 2020; Available at: https://www.aamc.org/eras-statistics-2019. Accessed April 10, 2020.
3. The MATCH National Resident Matching Program: 2020 Main Residency Match Results: Advanced Data Tables. 2020. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/03/Advance-Data-Tables-2020.pdf. Accessed April 10, 2020.
4. Huang BY, Hicks TD, Haidar GM, Pounds LL, Davies MG. An evaluation of the availability, accessibility, and quality of online content of vascular surgery training program websites for residency and fellowship applicants. J Vasc Surg. 2017;66:1892-901.
5. Miller VM, Padilla LA, Schuh A, Mauchley D, Cleveland D, Aburjania Z, Dabal R. Evaluation of cardiothoracic surgery residency and fellowship program websites. J Surg Res. 2019;246:200-6.
6. Skovrlj B, Silvestre J, Ibeh C, Abbatematteo JM, Mocco J. Neurosurgery residency websites: a critical evaluation. World Neurosurg. 2015;84:727-33.
7. Patel SJ, Abdullah MS, Yeh PC, Abdullah Z, Jayaram P. Content evaluation of physical medicine and rehabilitation residency websites. PM R. 2019; doi:10.1002/pmrj.12303.
8. Rozental TD, Lonner JH, Parekh SG. The internet as a communication tool for academic orthopaedic surgery departments in the United States. J Bone Joint Surg Am 2001;83:987-91.
9. Hinds RM, Danna NR, Capo JT, Mroczek KJ. Foot and ankle fellowship websites: an assessment of accessibility and quality. Foot Ankle Spec. 2017;10:302-7.
10. Hinds RM, Capo JT, Egol KA. Information on orthopedic trauma fellowships: online accessibility and content. Am J Orthop. 2017;46:E320-5.
11. Hinds RM, Klifto CS, Naik AA, Sapienza A, Capo JT. Hand society and matching program web sites provide poor access to information regarding hand surgery fellowship. J Hand Microsurg. 2016;8:91-5.
12. Gu A, Lehman JD, Sardana A, Cohen JS, Richardson SS, Sculco PK. Adult reconstruction hip and knee fellowship program content and accessibility. J Arthroplasty. 2018;33:1630-3.
13. Mulcahey MK, Gosselin MM, Fadale PD. Evaluation of the content and accessibility of web sites for accredited orthopaedic sports medicine fellowships. J Bone Joint Surg Am. 2013;95:e85.
14. Yayac M, Javandal M, Mulcahey MK. Accredited orthopaedic sports medicine fellowship websites: an updated assessment of accessibility and content. Orthop J Sports Med. 2017;5:2325967116683942.
15. Shaath MK, Yeranosian MG, Ippolito JA, Adams MR, Sirkin MS, Reilly MC. Evaluation of the content and accessibility of web sites for accredited orthopaedic trauma surgery fellowships. J Bone Joint Surg Am. 2018;100:e60.
16. Silvestre J, Guzman JZ, Skovrlj B, Overley SC, Cho SK, Qureshi SA, Hecht AC. The internet as a communication tool for orthopedic spine fellowships in the United States. Spine J. 2015;15:655-61.
17. Silvestre J, Guzman JZ, Abbatematteo JM, Chang B, Levin LS. Evaluation of content and accessibility of hand fellowship websites. Hand (N Y) 2015;10:516-21.
18. Young BL, Cantrell CK, Patt JC, Ponce BA. Accessibility and content of individualized adult reconstructive hip and knee/musculoskeletal oncology fellowship web sites. Arthroplast Today. 2018;4:232-5.
19. Young BL, Cantrell CK, Ponce BA, Quade JH. Evaluation of orthopaedic trauma fellowship web sites and an assessment of 21 content domains. Iowa Orthop J. 2018;38:39-43.
20. Young BL, Oladeji LO, Cichos K, Ponce B. Content and accessibility of shoulder and elbow fellowship web sites in the United States. Iowa Orthop J. 2016;36:36-40.
21. Trehan SK, Morrell NT, Akelman E. Accredited hand surgery fellowship Web sites: analysis of content and accessibility. J Hand Surg Am. 2015;40:778-82.
22. Davidson AR, Murphy RF, Spence DD, Kelly DM, Warner WC, Sawyer JR. Accessibility and quality of online information for pediatric orthopaedic surgery fellowships. J Pediatr Orthop. 2014;34:831-4.
23. Khwaja A, Du P, Sherman N, Truchan L. An update on foot and ankle fellowship website content and accessibility. Foot & Ankle Orthop. 2020;5:247301142091295.
24. Li NY, Gruppuso PA, Kalagara S, Eltorai AEM, DePasse JM, Daniels AH. Critical assessment of the contemporary orthopaedic surgery residency application process. J Bone Joint Surg Am. 2019;101:e114.
Copyright © 2020 The Authors. Published by The Journal of Bone and Joint Surgery, Incorporated. All rights reserved.