The single most valued resource used among residents is the Orthobullets (Copyright © 2018 Lineage Medical) platform, which consists of a website (https://www.orthobullets.com) and cell phone application, at 57.1% (212), textbooks at 22.6% (84), systematic review/meta-analysis/general review articles at 11% (41), and primary, focused research articles at 5.9% (22). Although 67.4% of residents use industry-sponsored events, no respondents find these events to be the single most valued resource.
In terms of journal resources used by residents, the single most valued is general review articles at 55.5% (206), followed by systematic review articles at 22.4% (83), and primary, focused research articles at 18.3% (68).
Table 2 displays a complete list of the educational resource options provided to residents as well as the percentage of residents who use the resource. Residents identify their top three most used resources as: (1) Orthobullets (236, 63.6%), (2) textbooks (111, 29.9%), and (3) systematic review/meta-analysis (76, 20.5%). In terms of preparation for a case or specific procedure, the top three resources differed, with the approach being to: (1) read a general website, like Orthobullets (197, 53.1%), (2) watch a relevant surgical video (116, 31.3%), and (3) read a relevant textbook excerpt (87, 23.5%).
Residents were asked to identify their top three online resources used to stay current and enhance fundamental orthopaedic knowledge. The responses were: (1) Orthobullets (345, 93%), (2) VuMedi (165, 44.5%), and (3) various websites (81, 21.8%). The approach to enhancing general knowledge is to prioritize general websites, like Orthobullets, and then move on to relevant textbooks, followed by relevant systematic reviews, meta-analyses, and general review articles.
The importance of scholarly journals was addressed by comparing several journals. Residents find the Journal of the American Academy of Orthopaedic Surgeons (JAAOS) and the Journal of Bone and Joint Surgery to be very important (200, 53.9% and 138, 37.2%, respectively). Specialty journals and Clinical Orthopedics and Related Research (CORR) were used but not deemed as important as educational resources. Specialty journals were seen as very important by 22% and CORR was viewed as very important by 8.4%.
No significant difference is evident between sexes in the mean amount of time spent studying per week (P = 0.94), with male respondents spending 10.7 hours and female respondents spending 10.8 hours. In addition, both sexes identify the single most valued journal resource as general review articles (55.9% of male respondents and 59.0% of female respondents) and the most valued resource in general as Orthobullets (58.7% of male respondents and 50.8% of female respondents).
Specific resources are not used markedly differently based on sex, as seen in Table 3. High percentages of each group use Orthobullets, textbooks, systematic review/meta-analyses, primary, focused articles, and VuMedi. No difference between sexes exist in terms of how they use industry-sponsored events, with >50% of male and female residents doing so. Finally, no notable difference exists in how male and female residents prioritize their top three resources in general or in how they prioritize resources while specifically preparing for a case.
Male and female residents significantly differed in their assessment of the importance of scholarly journals. More female residents find the CORR journal to be very important or important (11.5%, 44.3%, respectively, for females; 7.5%, 28.7%, respectively, for males with a P = 0.004). The same occurred for specialty journals, with a larger percentage of female residents finding them to be very important or important (32.8%, 42.6%, respectively, for females; 19.9%, 36.5%, respectively, for males with a P = 0.016).
Finally, a significantly greater percentage of female residents strongly agreed (27.9%) or agreed (44.3%) that the addition of specific research articles to the JAAOS increases the educational value of the journal, versus 21.2% of male residents who strongly agreed and 32.6% who agreed (P = 0.007).
No significant difference is observed between location of resident in mean amount of time spent studying per week (P = 0.72), with residents from the Midwest spending 10.5 hours, Northeast spending 11.3 hours, South spending 10.2 hours, and West spending 10.8 hours. In addition, residents from all locations feel that the single most valued journal resource is general review articles, 44.3% to 64.4% (P = 0.032). However, the Midwest, Northeast, and South emphasize primary, focused articles (17.8% to 23.4%) more than the West (12%) and the Northeast, South, and West prefer systematic review articles (20.0% to 32.1%) more than the Midwest (16.1%, P = 0.032). Residents from all over the country use similar electronic resources, including Orthobullets, VuMedi, YouTube, and various websites. Orthobullets is the primary resource used in all locations.
Year in Training
There was no significant difference in the mean amount of time spent studying per week based on training level (ie, postgraduate year [PGY]), with PGY-1 spending 10.6, PGY-2 spending 12.1, PGY-3 spending 10.3, PGY-4 spending 10.6, and PGY-5/6 level spending 9.9 hours (P = 0.51). Residents in all the PGYs of training report the single most valuable resource as Orthobullets, followed by textbooks. Significant differences were found in VuMedi use, which steadily increased each year of training until PGY-4 (P < 0.001). Similarly, residents earlier in training appear to use YouTube at a greater rate (53% of PGY-1 residents, 41% of PGY-2 residents) than those at a higher level of training, with only 23% of trainees at the PGY-5/6 level (P = 0.026). Data can be found in Table 4.
A similar trend was seen when residents were asked whether they used online resources to enhance their fundamental knowledge, with residents earlier in training using YouTube more than senior residents. Sixty-five percent of PGY-1 level and 45% of PGY-2 level residents use YouTube, whereas only 25% of PGY-5/6 levels do so (P = 0.001). Senior residents use VuMedi more than juniors, with 60% of PGY-5/6 level residents endorsing use and only 38% of PGY-1 level residents (P = 0.001).
As residents progress in training, a significant increase in prioritizing primary, focused research articles (P = 0.013) is observed. Although this resource was never ranked as the top resource, a greater percentage of residents reported using it as they rose in training level.
This study provides insight into the manner in which residents are currently being educated as well as how they prioritize and invest their time. Since the initiation of work-hour restrictions on July 1, 2003, residents are forced to assimilate the same mass of knowledge.11-13 They must do so in an expeditious fashion in a changing academic setting, given that surgical volume is unchanged for junior residents and increased for senior residents after hour regulations.14 Higher educational learning is changing with evolving technology and instant access to unlimited resources, as well as a large volume of yearly publications in the field.15 More than 600,000 articles are published in biomedical literature every year, providing an insurmountable mountain of changing medical literature.15 Resident physicians are clearly users of technology and have been raised in a generation in which healthcare is available on Internet platforms and resources are electronically at their fingertips.16
Electronic platforms, such as Orthobullets, appear to have advanced the way residents learn by providing an easily accessible, free gateway to organized information about the field. Our data revealed the overwhelming prioritization of this resource as supplement to both general knowledge and in case preparation. With immediate access to information about various topics in orthopaedics, either online or through a cell phone application, residents are able to learn about everything from the basics to specific surgical approaches. Residents almost universally use Orthobullets—as was seen with the 99.5% in our data set—whereas systematic review/meta-analysis, textbooks, and primary, focused articles follow behind. In addition, all residents in the country, regardless of year of training or sex, find value in this resource. This is departure from a time when journals or textbooks served as the mainstay of educational resources. Instead, a new generation of digital natives who consume, prioritize, and seek information quite differently is observed.16,17 It is also important to note that open-access resources do not always have a clearly stated review process that systematically removes errors and verifies accuracy. Many websites and various electronic resources are not subject to the peer review process of journal submissions or textbook authorship.
The role of industry-sponsored events and learning is also an important topic. A notable 67.4% of residents report using industry-sponsored events for education. However, not a single resident felt this to be the single most valued resource. This speaks to how industry-sponsored events are used: quite frequently, but not as the prioritized resource. Industry-sponsored events provide a time outside of the operating room to become familiar with instruments and techniques and are especially beneficial for those earlier in training.18 Industry influence remains a controversial topic and a source of bias that the resident must acknowledge and attempt to control.19 The role of industry in our field is a large one, evidenced by the thousands of industry-sponsored training courses and residency curriculums that include interaction between device representatives and various implants and tools.19 There is also focus on the ethics surrounding the relationship between the surgeon and device representative.20 In addition, it has been established that orthopaedic surgeons and residents do not understand the trust cost and cost variability of various devices and implants.21 Given this, it is encouraging that orthopaedic residents are using industry-sponsored education opportunities but not focusing on them as the backbone of their study approach.
Reaching residents was a very difficult aspect of the study. There was no centralized way to access all residents, despite the existence of larger, membership-driven societies and governing bodies representing the field. Contacting participants required coordination between various program representatives, and a larger pool of residents might have been captured if a more centralized form of communication or contact list were available. It was not possible to calculate response rate because we did not know how many residents received the survey. Using the approximate number of residents in the country (3,889),2 and our 374 responses, this provides a bare minimum response rate of 9.6%. Realistically, it was higher than this because we did not reach all residents with the survey.
Of note, respondents were 16% female, years in training were evenly distributed in respondents, and the data represented programs across the country. Given that 14% of orthopaedic residents were female in 2016 to 2017,22,23 our respondents appear to be in line with national percentages, providing support that our data are representative. It is still a concern that our data will not be generalizable, however.
With any survey study, response bias is always possible, that is, the responders may differ systematically from the nonresponders. We were not able to comprehensively compare demographics between responders and nonresponders. Finally, potential for misunderstanding or other response errors associated with survey instruments is always present. There is also the possibility that bias was introduced by the electronic nature of the survey, which could have been preferentially completed by respondents who use electronic resources more often.
Orthopaedic surgical residents are attempting to consume the same amount of information and garner the same set of surgical skills as previous residents. However, they do so in a new age of technology with a variety of electronic educational resources and with externally imposed time constraints. Current residents—regardless of sex, location, and year in training—use Orthobullets and other electronic resources as the mainstay of their fund of orthopaedic knowledge as well as in case preparation. There continues to be utilization of industry-sponsored resources, although they are not the primary focus for those in training.
Journals such as JAAOS and Journal of Bone and Joint Surgery continue to play a role in resident study. There are subtle, notable differences in terms of the sexes, with female residents looking more favorably on the addition of research articles to JAAOS. In addition, specialty-focused journals and research-focused journals are not viewed as important by residents. Regionally, residents are similar in terms of time committed to study and most valued resources. The single most valued journal resource is general review articles throughout the country, although there are regional differences with the Midwest, Northeast, and South using primary, focused articles and the Northeast, South, and West choosing systematic review articles.
Understanding which resources are emphasized in training is important not only for residents but also for program directors and faculty because they may choose to tailor the way in which they set expectations, provide access to resources, and assess resident progress. Given the large consumption of free access material by residents, it raises the question of whether a process for consensus on accuracy is warranted, as well as whether the orthopaedic community as a whole should develop streamlined and reviewed platforms through which to foster orthopaedic surgeons in training.
1. Egol KA, Dirschl DR, Levine WN, Zuckerman JD: Orthopaedic residency education: A practical guide to selection, training, and education. AAOS Instr Course Lect 2013;62:553–564.
2. Brotherton SE, Etzel SI: Graduate medical education, 2017-2018. JAMA 2018;320:1051–1070.
4. Wolf BR, Britton CL: How orthopaedic residents perceive educational resources. Iowa Orthop J 2013;33:185–190.
5. Tarpada SP, Morris MT, Burton DA: E-learning in orthopedic surgery training: A systematic review. J Orthop 2016;13:425–430.
6. Edson RS, Beckman TJ, West CP, et al: A multi-institutional survey of internal medicine residents' learning habits. Med Teach 2010;32:773–775.
7. Davis JS, Garcia GD, Wyckoff MM, et al: Knowledge and usability of a trauma training system for general surgery residents. Am J Surg 2013;205:681–684.
8. Satterwhite T, Son J, Carey J, et al: Microsurgery education in residency training: Validating an online curriculum. Ann Plast Surg 2012;68:410–414.
9. Burke DT, Judelson AL, Schneider JC, DeVito MC, Latta D: Reading habits of practicing physiatrists. Am J Phys Med Rehabil 2002;81:779–787.
10. Schein M, Paladugu R, Sutija VG, Wise L: What American surgeons read: A survey of a thousand fellows of the American College of Surgeons. Curr Surg 2000;57:252–258.
11. Jack MC, Kenkare SB, Saville BR, et al: Improving education under work-hour restrictions: Comparing learning and teaching preferences of faculty, residents, and students. J Surg Educ 2010;67:290–296.
12. Mauser NS, Michelson JD, Gissel H, Henderson C, Mauffrey C: Work-hour restrictions and orthopaedic resident education: A systematic review. Int Orthop 2015;40:865–873.
13. Zuckerman JD, Kubiak EN, Immerman I, Dicesare P: The early effects of code 405 work rules on attitudes of orthopaedic residents and attending surgeons. J Bone Joint Surg Am 2005;87:903–908.
14. Baskies MA, Ruchelsman DE, Capeci CM, Zuckerman JD, Egol KA: Operative experience in an orthopaedic surgery residency program: The effect of work-hour restrictions. J Bone Joint Surg Am 2008;90;924–927.
15. Davidson AR, Loftis CM, Throckmorton TW, Kelly DM: Accessibility and availability of online information for orthopedic surgery residency programs. Iowa Orthop J 2016;36:31–36.
16. Parekh SG, Nazarian DG, Lim CK: Adoption of information technology by resident physicians. Clin Orthop Relat Res 2004;421:107–111.
18. Stirling ERB, Lewis TL, Ferran NA: Surgical skills simulation in trauma and orthopaedic training. J Orthop Surg Res 2014;9:1–9.
19. Sarmiento A: On the education of the orthopaedic resident. Indian J Orthop 2008;42:241–244.
20. O'Connor B, Pollner F, Fugh-Berman A: Salespeople in the surgical suite: Relationships between surgeons and medical device representatives. PLoS One 2016;11:e0158510.
21. Okike K, O'Toole RV, Pollak AN, et al: Survey finds few orthopedic surgeons know the costs of the devices they implant. Health Aff (Millwood) 2014;33:103–109.
23. Chambers CC, Ihnow SB, Monroe EJ, Suleiman LI: Women in orthopaedic surgery: Population trends in trainees and practicing surgeons. J Bone Joint Surg Am 2018;100:e116.
Supplemental Digital Content
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the American Academy of Orthopaedic Surgeons