Secondary Logo

Educational Resource Utilization by Current Orthopaedic Surgical Residents

A Nation-wide Survey

Rogers, Miranda J., MS, MD; Zeidan, Michelle, MD; Flinders, Zachary S., BS; Presson, Angela P., PhD; Burks, Robert, MD

JAAOS Global Research & Reviews: April 2019 - Volume 3 - Issue 4 - p e041
doi: 10.5435/JAAOSGlobal-D-19-00041
Research Article

Background: More than 150 accredited orthopaedic residency programs exist in the United States with more than 3,300 residents. Every year, between 600 and 800 orthopaedic surgeons complete board certification. However, little is known about what residents use and prefer of the many resources available to them. The purpose of this study is to understand how orthopaedic residents across the country prioritize educational resources.

Methods: An 18-question survey with subqueries was compiled on the REDCap (Research Electronic Data Capture) platform. Residents were reached through program coordinators, program directors, and word of mouth. There were 374 total respondents. Responses were summarized for the total population and stratified separately by sex, location, and year of training. P values were adjusted for multiple comparisons using the Bonferroni method.

Results: Respondents were 83.4% male and 16.6% female. Orthobullets was the single most valued resource and general review articles were the single most valued journal resource. The average time spent studying was 10.7 hr/wk (SD = 7.6), with no notable differences by program location or sex. A significantly larger percentage of female residents placed more importance on specialty-specific journals (32.8% rated as very important versus 19.9% from males, P = 0.016) and research-related journals (P = 0.004). VuMedi use significantly increased until postgraduate year-4 (P < 0.001), whereas residents earlier in training used YouTube more often (P = 0.026). There were differences in journal prioritization by program region, with the Midwest, Northeast, and South preferring primary, focused articles and the Northeast, South, and West emphasizing systematic review articles (P = 0.032). Industry-sponsored events were used by 67.4% of residents as an educational adjunct.

Conclusions: Current residents—regardless of sex, location, and year in training—use Orthobullets and other electronic resources. There continues to be utilization of industry-sponsored resources, although they are not the primary focus for those in training.

From the Department of Orthopaedic Surgery, University of Utah Orthopedic Center, University of Utah, Salt Lake City, UT (Dr. Rogers, Dr. Zeidan, and Dr. Burks); the University of Utah School of Medicine, Salt Lake City, UT (Mr. Flinders); and the Division of Epidemiology, Department of Orthopaedic Surgery, University of Utah, Salt Lake City, UT (Dr. Presson).

Correspondence to Dr. Rogers:

Investigation supported by the University of Utah Population Health Research Foundation, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health (Grant 5UL1TR001067-05).

None of the following authors or any immediate family member has received anything of value from or has stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this article: Dr. Rogers, Dr. Zeidan, Mr. Flinders, Dr. Presson, and Dr. Burks.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Understanding modern orthopaedic residency education paradigms as well as how residents identify, consume, and prioritize resources is critical for maintaining an effective orthopaedic residency program. Most agree that the education of trainees falls upon all those involved in a residency.1 There are approximately 154 accredited orthopaedic residency programs in the United States and 3,889 residents.2 Every year since 1934, between 600 and 800 orthopaedic surgeons complete board certification through the American Board of Orthopaedic Surgery.3 Despite this, no standardized way is observed in which orthopaedic residents are educated between the time they enter residency and the time they graduate.4 Resources available to residents include didactic lectures, clinical settings, operating room experience, texts and journals, and electronic and web-based platforms.4 In addition, many programs facilitate access to surgical skills laboratories, simulators, cadaveric dissection, training courses, and industry-funded events and materials.4 In recent years, electronic resources have become mainstay sources used by residents to gain knowledge and successfully navigate examinations.5 This pattern of electronic resource utilization is seen throughout medicine6 and in surgical training programs7 and clearly represents an evolving trend in how residents learn. In addition, electronic resources have been shown to reduce error rate and surgical time, as well as improve surgical skill.5,7,8

Although there has been some research on resident resource utilization in orthopaedic surgery,5 there is still much to be understood about how resources are prioritized by learners in programs across the country during a time when electronic healthcare resources have become mainstream. The goals of this study are to: (1) determine the primary resources chosen by orthopaedic residents as well as how they are used; (2) identify regional differences in resource use in orthopaedic surgery programs; and to (3) assess for differences in sex in terms of resource utilization.

Back to Top | Article Outline


An anonymous 18-question survey with subqueries was developed based on surveys used in similar studies9,10 and constructed in REDCap (Research Electronic Data Capture) (Appendix A, Residency coordinators were contacted and then distributed the survey to their program. We also used word of mouth between any combination of residents, coordinators, and directors. There was no central database available through which to contact residents, and we could not verify the total number of residents who ultimately received the survey.

There were 374 total respondents from October 2017 to March 2018, with three subjects excluded because their year of training was listed as other or missing. Year of training 5 and 6 were combined together. Survey responses were summarized for the total population and stratified separately by sex, location, and year of training. We used a two-sample t-test to compare hours per week spent researching between male and female respondents and a one-way ANOVA (analysis of variance) test to compare weekly hours spent studying across program locations and years of training. Responses of yes/no and other nonordered responses were analyzed using chi squared or Fisher's exact tests. Ordered responses were compared with sex using Wilcoxon rank-sum tests and with program location and year of training using Kruskal-Wallis equality-of-populations rank tests. Data were analyzed using Stata 15 (Stata Statistical Software: Release 15; StataCorp LP). For each survey question that queried a use of multiple educational resources, we used a Bonferroni adjustment for multiple comparisons (ie, we multiplied P-values by the number of tests conducted). Statistical significance was assessed at the 0.05 level, and all tests were two-tailed.

Back to Top | Article Outline



Of the respondents, 83.4% are male (307), 16.0% are female (61), and three preferred not to specify sex (<1%). The mean hours spent studying per week is 10.7, with a SD of 7.65. Years of training and geographic location were evenly represented, and demographic data is given in Table 1 and Figure 1.

Table 1

Table 1

Figure 1

Figure 1

The single most valued resource used among residents is the Orthobullets (Copyright © 2018 Lineage Medical) platform, which consists of a website ( and cell phone application, at 57.1% (212), textbooks at 22.6% (84), systematic review/meta-analysis/general review articles at 11% (41), and primary, focused research articles at 5.9% (22). Although 67.4% of residents use industry-sponsored events, no respondents find these events to be the single most valued resource.

In terms of journal resources used by residents, the single most valued is general review articles at 55.5% (206), followed by systematic review articles at 22.4% (83), and primary, focused research articles at 18.3% (68).

Table 2 displays a complete list of the educational resource options provided to residents as well as the percentage of residents who use the resource. Residents identify their top three most used resources as: (1) Orthobullets (236, 63.6%), (2) textbooks (111, 29.9%), and (3) systematic review/meta-analysis (76, 20.5%). In terms of preparation for a case or specific procedure, the top three resources differed, with the approach being to: (1) read a general website, like Orthobullets (197, 53.1%), (2) watch a relevant surgical video (116, 31.3%), and (3) read a relevant textbook excerpt (87, 23.5%).

Table 2

Table 2

Residents were asked to identify their top three online resources used to stay current and enhance fundamental orthopaedic knowledge. The responses were: (1) Orthobullets (345, 93%), (2) VuMedi (165, 44.5%), and (3) various websites (81, 21.8%). The approach to enhancing general knowledge is to prioritize general websites, like Orthobullets, and then move on to relevant textbooks, followed by relevant systematic reviews, meta-analyses, and general review articles.

The importance of scholarly journals was addressed by comparing several journals. Residents find the Journal of the American Academy of Orthopaedic Surgeons (JAAOS) and the Journal of Bone and Joint Surgery to be very important (200, 53.9% and 138, 37.2%, respectively). Specialty journals and Clinical Orthopedics and Related Research (CORR) were used but not deemed as important as educational resources. Specialty journals were seen as very important by 22% and CORR was viewed as very important by 8.4%.

Back to Top | Article Outline


No significant difference is evident between sexes in the mean amount of time spent studying per week (P = 0.94), with male respondents spending 10.7 hours and female respondents spending 10.8 hours. In addition, both sexes identify the single most valued journal resource as general review articles (55.9% of male respondents and 59.0% of female respondents) and the most valued resource in general as Orthobullets (58.7% of male respondents and 50.8% of female respondents).

Specific resources are not used markedly differently based on sex, as seen in Table 3. High percentages of each group use Orthobullets, textbooks, systematic review/meta-analyses, primary, focused articles, and VuMedi. No difference between sexes exist in terms of how they use industry-sponsored events, with >50% of male and female residents doing so. Finally, no notable difference exists in how male and female residents prioritize their top three resources in general or in how they prioritize resources while specifically preparing for a case.

Table 3

Table 3

Male and female residents significantly differed in their assessment of the importance of scholarly journals. More female residents find the CORR journal to be very important or important (11.5%, 44.3%, respectively, for females; 7.5%, 28.7%, respectively, for males with a P = 0.004). The same occurred for specialty journals, with a larger percentage of female residents finding them to be very important or important (32.8%, 42.6%, respectively, for females; 19.9%, 36.5%, respectively, for males with a P = 0.016).

Finally, a significantly greater percentage of female residents strongly agreed (27.9%) or agreed (44.3%) that the addition of specific research articles to the JAAOS increases the educational value of the journal, versus 21.2% of male residents who strongly agreed and 32.6% who agreed (P = 0.007).

Back to Top | Article Outline


No significant difference is observed between location of resident in mean amount of time spent studying per week (P = 0.72), with residents from the Midwest spending 10.5 hours, Northeast spending 11.3 hours, South spending 10.2 hours, and West spending 10.8 hours. In addition, residents from all locations feel that the single most valued journal resource is general review articles, 44.3% to 64.4% (P = 0.032). However, the Midwest, Northeast, and South emphasize primary, focused articles (17.8% to 23.4%) more than the West (12%) and the Northeast, South, and West prefer systematic review articles (20.0% to 32.1%) more than the Midwest (16.1%, P = 0.032). Residents from all over the country use similar electronic resources, including Orthobullets, VuMedi, YouTube, and various websites. Orthobullets is the primary resource used in all locations.

Back to Top | Article Outline

Year in Training

There was no significant difference in the mean amount of time spent studying per week based on training level (ie, postgraduate year [PGY]), with PGY-1 spending 10.6, PGY-2 spending 12.1, PGY-3 spending 10.3, PGY-4 spending 10.6, and PGY-5/6 level spending 9.9 hours (P = 0.51). Residents in all the PGYs of training report the single most valuable resource as Orthobullets, followed by textbooks. Significant differences were found in VuMedi use, which steadily increased each year of training until PGY-4 (P < 0.001). Similarly, residents earlier in training appear to use YouTube at a greater rate (53% of PGY-1 residents, 41% of PGY-2 residents) than those at a higher level of training, with only 23% of trainees at the PGY-5/6 level (P = 0.026). Data can be found in Table 4.

Table 4

Table 4

A similar trend was seen when residents were asked whether they used online resources to enhance their fundamental knowledge, with residents earlier in training using YouTube more than senior residents. Sixty-five percent of PGY-1 level and 45% of PGY-2 level residents use YouTube, whereas only 25% of PGY-5/6 levels do so (P = 0.001). Senior residents use VuMedi more than juniors, with 60% of PGY-5/6 level residents endorsing use and only 38% of PGY-1 level residents (P = 0.001).

As residents progress in training, a significant increase in prioritizing primary, focused research articles (P = 0.013) is observed. Although this resource was never ranked as the top resource, a greater percentage of residents reported using it as they rose in training level.

Back to Top | Article Outline


This study provides insight into the manner in which residents are currently being educated as well as how they prioritize and invest their time. Since the initiation of work-hour restrictions on July 1, 2003, residents are forced to assimilate the same mass of knowledge.11-13 They must do so in an expeditious fashion in a changing academic setting, given that surgical volume is unchanged for junior residents and increased for senior residents after hour regulations.14 Higher educational learning is changing with evolving technology and instant access to unlimited resources, as well as a large volume of yearly publications in the field.15 More than 600,000 articles are published in biomedical literature every year, providing an insurmountable mountain of changing medical literature.15 Resident physicians are clearly users of technology and have been raised in a generation in which healthcare is available on Internet platforms and resources are electronically at their fingertips.16

Electronic platforms, such as Orthobullets, appear to have advanced the way residents learn by providing an easily accessible, free gateway to organized information about the field. Our data revealed the overwhelming prioritization of this resource as supplement to both general knowledge and in case preparation. With immediate access to information about various topics in orthopaedics, either online or through a cell phone application, residents are able to learn about everything from the basics to specific surgical approaches. Residents almost universally use Orthobullets—as was seen with the 99.5% in our data set—whereas systematic review/meta-analysis, textbooks, and primary, focused articles follow behind. In addition, all residents in the country, regardless of year of training or sex, find value in this resource. This is departure from a time when journals or textbooks served as the mainstay of educational resources. Instead, a new generation of digital natives who consume, prioritize, and seek information quite differently is observed.16,17 It is also important to note that open-access resources do not always have a clearly stated review process that systematically removes errors and verifies accuracy. Many websites and various electronic resources are not subject to the peer review process of journal submissions or textbook authorship.

The role of industry-sponsored events and learning is also an important topic. A notable 67.4% of residents report using industry-sponsored events for education. However, not a single resident felt this to be the single most valued resource. This speaks to how industry-sponsored events are used: quite frequently, but not as the prioritized resource. Industry-sponsored events provide a time outside of the operating room to become familiar with instruments and techniques and are especially beneficial for those earlier in training.18 Industry influence remains a controversial topic and a source of bias that the resident must acknowledge and attempt to control.19 The role of industry in our field is a large one, evidenced by the thousands of industry-sponsored training courses and residency curriculums that include interaction between device representatives and various implants and tools.19 There is also focus on the ethics surrounding the relationship between the surgeon and device representative.20 In addition, it has been established that orthopaedic surgeons and residents do not understand the trust cost and cost variability of various devices and implants.21 Given this, it is encouraging that orthopaedic residents are using industry-sponsored education opportunities but not focusing on them as the backbone of their study approach.

Reaching residents was a very difficult aspect of the study. There was no centralized way to access all residents, despite the existence of larger, membership-driven societies and governing bodies representing the field. Contacting participants required coordination between various program representatives, and a larger pool of residents might have been captured if a more centralized form of communication or contact list were available. It was not possible to calculate response rate because we did not know how many residents received the survey. Using the approximate number of residents in the country (3,889),2 and our 374 responses, this provides a bare minimum response rate of 9.6%. Realistically, it was higher than this because we did not reach all residents with the survey.

Of note, respondents were 16% female, years in training were evenly distributed in respondents, and the data represented programs across the country. Given that 14% of orthopaedic residents were female in 2016 to 2017,22,23 our respondents appear to be in line with national percentages, providing support that our data are representative. It is still a concern that our data will not be generalizable, however.

With any survey study, response bias is always possible, that is, the responders may differ systematically from the nonresponders. We were not able to comprehensively compare demographics between responders and nonresponders. Finally, potential for misunderstanding or other response errors associated with survey instruments is always present. There is also the possibility that bias was introduced by the electronic nature of the survey, which could have been preferentially completed by respondents who use electronic resources more often.

Back to Top | Article Outline


Orthopaedic surgical residents are attempting to consume the same amount of information and garner the same set of surgical skills as previous residents. However, they do so in a new age of technology with a variety of electronic educational resources and with externally imposed time constraints. Current residents—regardless of sex, location, and year in training—use Orthobullets and other electronic resources as the mainstay of their fund of orthopaedic knowledge as well as in case preparation. There continues to be utilization of industry-sponsored resources, although they are not the primary focus for those in training.

Journals such as JAAOS and Journal of Bone and Joint Surgery continue to play a role in resident study. There are subtle, notable differences in terms of the sexes, with female residents looking more favorably on the addition of research articles to JAAOS. In addition, specialty-focused journals and research-focused journals are not viewed as important by residents. Regionally, residents are similar in terms of time committed to study and most valued resources. The single most valued journal resource is general review articles throughout the country, although there are regional differences with the Midwest, Northeast, and South using primary, focused articles and the Northeast, South, and West choosing systematic review articles.

Understanding which resources are emphasized in training is important not only for residents but also for program directors and faculty because they may choose to tailor the way in which they set expectations, provide access to resources, and assess resident progress. Given the large consumption of free access material by residents, it raises the question of whether a process for consensus on accuracy is warranted, as well as whether the orthopaedic community as a whole should develop streamlined and reviewed platforms through which to foster orthopaedic surgeons in training.

Back to Top | Article Outline


1. Egol KA, Dirschl DR, Levine WN, Zuckerman JD: Orthopaedic residency education: A practical guide to selection, training, and education. AAOS Instr Course Lect 2013;62:553–564.
2. Brotherton SE, Etzel SI: Graduate medical education, 2017-2018. JAMA 2018;320:1051–1070.
3. Martin DF: President's Message [Internet]. American Board of Orthopaedic Surgery, The Diplomate ENewsletter, 2017. Accessed September 12, 2018.
4. Wolf BR, Britton CL: How orthopaedic residents perceive educational resources. Iowa Orthop J 2013;33:185–190.
5. Tarpada SP, Morris MT, Burton DA: E-learning in orthopedic surgery training: A systematic review. J Orthop 2016;13:425–430.
6. Edson RS, Beckman TJ, West CP, et al: A multi-institutional survey of internal medicine residents' learning habits. Med Teach 2010;32:773–775.
7. Davis JS, Garcia GD, Wyckoff MM, et al: Knowledge and usability of a trauma training system for general surgery residents. Am J Surg 2013;205:681–684.
8. Satterwhite T, Son J, Carey J, et al: Microsurgery education in residency training: Validating an online curriculum. Ann Plast Surg 2012;68:410–414.
9. Burke DT, Judelson AL, Schneider JC, DeVito MC, Latta D: Reading habits of practicing physiatrists. Am J Phys Med Rehabil 2002;81:779–787.
10. Schein M, Paladugu R, Sutija VG, Wise L: What American surgeons read: A survey of a thousand fellows of the American College of Surgeons. Curr Surg 2000;57:252–258.
11. Jack MC, Kenkare SB, Saville BR, et al: Improving education under work-hour restrictions: Comparing learning and teaching preferences of faculty, residents, and students. J Surg Educ 2010;67:290–296.
12. Mauser NS, Michelson JD, Gissel H, Henderson C, Mauffrey C: Work-hour restrictions and orthopaedic resident education: A systematic review. Int Orthop 2015;40:865–873.
13. Zuckerman JD, Kubiak EN, Immerman I, Dicesare P: The early effects of code 405 work rules on attitudes of orthopaedic residents and attending surgeons. J Bone Joint Surg Am 2005;87:903–908.
14. Baskies MA, Ruchelsman DE, Capeci CM, Zuckerman JD, Egol KA: Operative experience in an orthopaedic surgery residency program: The effect of work-hour restrictions. J Bone Joint Surg Am 2008;90;924–927.
15. Davidson AR, Loftis CM, Throckmorton TW, Kelly DM: Accessibility and availability of online information for orthopedic surgery residency programs. Iowa Orthop J 2016;36:31–36.
16. Parekh SG, Nazarian DG, Lim CK: Adoption of information technology by resident physicians. Clin Orthop Relat Res 2004;421:107–111.
17. Prensky M: Digital natives, digital immigrants [Internet]. On the Horizon, MCB University Press, 2001.,%20Digital%20Immigrants%20-%20Part1.pdf. Accessed September 8, 2018.
18. Stirling ERB, Lewis TL, Ferran NA: Surgical skills simulation in trauma and orthopaedic training. J Orthop Surg Res 2014;9:1–9.
19. Sarmiento A: On the education of the orthopaedic resident. Indian J Orthop 2008;42:241–244.
20. O'Connor B, Pollner F, Fugh-Berman A: Salespeople in the surgical suite: Relationships between surgeons and medical device representatives. PLoS One 2016;11:e0158510.
21. Okike K, O'Toole RV, Pollak AN, et al: Survey finds few orthopedic surgeons know the costs of the devices they implant. Health Aff (Millwood) 2014;33:103–109.
22. Harrison L: Women still find doors barred to orthopedic residencies [Internet]. Medscape 2018. Accessed September 10, 2018.
23. Chambers CC, Ihnow SB, Monroe EJ, Suleiman LI: Women in orthopaedic surgery: Population trends in trainees and practicing surgeons. J Bone Joint Surg Am 2018;100:e116.

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the American Academy of Orthopaedic Surgeons