Journal Logo

Scientific Articles

Availability and Quality of Online Information on Sub-Internships in U.S. Orthopaedic Residency Programs

Rai, Rahul BA1; Sabharwal, Sanjeev MD, MPH2,3

Author Information
doi: 10.2106/JBJS.OA.18.00036
  • Open
  • Disclosures


Orthopaedic surgery is widely considered one of the most competitive specialties for a fourth-year medical student to match into a residency position. In the 2018 National Residency Matching Program (NRMP), 171 Accreditation Council for Graduate Medical Education (ACGME)-approved orthopaedic surgery programs listed 742 PGY-1 (postgraduate year-1) positions1. There were 849 U.S. senior student applicants, and 4 unfilled positions, with an overall match rate of 81% for U.S. senior students1. It is well known that securing a position at an orthopaedic surgery residency program is a competitive process. Therefore, applicants put great effort into making their application as strong as possible in order to be accepted into a desirable residency position. However, as reported by Baldwin et al., the orthopaedic sub-internship or “away rotation” has recently become a critical factor in successful orthopaedic surgery residency applications2. Although that study was conducted in 2009, the study design assumed that only 75% of students would be enrolled in a sub-internship. In 2018, we can assume that nearly every applicant to orthopaedic surgery enrolls in a sub-internship and that, if the 2018 student demographics were applied to the 2009 study, the importance of the sub-internship would have been even more profound.

The sub-internship (also referred to as sub-I, acting internship, AI, away rotation) is typically a 4-week rotation that fourth-year medical students enroll in at an institution that is often different from their home institution. One of the primary goals of the sub-internship is to allow both the visiting student as well as the residents and faculty of the host program to assess the rotating applicant’s “fit” in the program on the basis of the student’s interpersonal skills, clinical aptitude, and ability to integrate into the program culture3. Thus, besides acquiring clinical skills and knowledge and assessing the appropriateness of the training from the student’s perspective, the sub-internship can be considered a 4-week-long audition during which the host institution can evaluate the desirability of having the applicant as a future resident in its training program. This acquaintance may allow residency program directors to separate and stratify applicants with greater precision when ranking applicants for the NRMP “Match.”3 In addition, while not guaranteed, the sub-internship often gives the student a greater chance at receiving an interview at that institution for a residency position.

While previous reports have examined the content and quality of online information pertaining to visiting students for residency program web sites for specialties such as pediatrics4, dermatology5, general surgery6, neurosurgery7, and otolaryngology8, there is little discussion in the current literature regarding the availability and quality of online information for orthopaedic sub-internships. The authors of those reports from other medical specialties came to the similar conclusion that the general quality and availability of information on residency program web sites was inadequate and lacking in quality when pertaining to medical student applicants. The purpose of the present study was to assess the availability and content of online information available for sub-internships in orthopaedics among U.S. allopathic orthopaedic residency program web sites.

Materials and Methods

We browsed the web sites of 151 orthopaedic residency programs on the Electronic Residency Application Service (ERAS) ( using the Google search engine on December 28, 2017. A second search was performed on February 23, 2018 to check for any updated information that may have been missed during the initial search. The search terms always followed the same format: “X orthopaedic surgery residency,” where “X” would be the name of the program as listed in ERAS. Each orthopaedic surgery residency program web site was assessed for the following 4 criteria: (1) any description or mention of a sub-internship offered by that program, (2) contact information regarding the sub-internship, (3) a list of learning objectives to be met by the rotating student during the sub-internship, and (4) a web page dedicated solely to the orthopaedic sub-internship offered by that program. Given the lack of any previously published quantifiable measure for grading online content of sub-internships, we allotted each orthopaedic program web site a non-weighted sub-internship score (SI score) of 0 to 4 depending on how many of the above criteria were met for the individual web site. Program web sites that did not have any information regarding sub-internships were given a score of 0, program web sites that met any 1 of the 4 criteria were given a score of 1, and so on. Programs were further stratified according to size, including the listed number of residents and full-time faculty, and according to the program’s geographic location (Northeast, South, Midwest, Southwest, and West). Programs also were categorized according to whether they were university-based or community-based. A university-based residency program was defined as one in which the main teaching hospital for postgraduate orthopaedic training was also the primary hospital affiliated with a medical school. A community-based program, on the other hand, was one in which the main teaching hospital was not the primary teaching hospital for a medical school.

All data were recorded with use of a specifically designed data-collection sheet by a single individual (R.R.). The collected information included the prevalence of available online information and the corresponding SI score. The data were further analyzed for possible relationships between the SI score and various demographic criteria such as the total number of residents, the number of full-time teaching faculty members in the department of orthopaedics, geographic location, and university-based versus community-based status of the program.

Statistical analysis was performed with use of SAS software (version 9.4; SAS Institute). The unpaired t test was used for continuous variables, and the chi-square test was used for categorical variables. SI scores were compared for various subgroups, including by university-based versus community-based programs and by the number of residents and the number of full-time teaching faculty. One-way analysis of variance (ANOVA) was used to compare SI scores of programs among the 5 geographic regions. Pearson correlation was used to assess the relationship between the number of residents or faculty and the program’s SI score. Univariate and multiple logistic regression analyses were utilized. The level of significance was set at p < 0.05.


A total of 151 orthopaedic surgery residency program web sites were assessed with use of the search terms described above. Of these 151program web pages, 69 (46%) had an SI score of 0, 4 (3%) had a score of 1, 18 (12%) had a score of 2, 20 (13%) had a score of 3, and 40 (26%) had a score of 4 (Table I).

TABLE I - Sub-Internship (SI) Scores Among Programs
SI Score No. of Programs
0 69 (46%)
1 4 (3%)
2 18 (12%)
3 20 (13%)
4 40 (26%)

Of the 151 programs, 42 were community-based programs and 109 were university-based programs. The average SI score was 1.05 for the community-based programs, compared with 1.98 for the university-based programs (p = 0.003) (Fig. 1). There was a weak association between the SI score and the number of residents in a given program (R2 = 0.074, p = 0.0004) (Fig. 2). There was no significant relationship between the SI score and the number of full-time faculty (R2 = 0.0045, p = 0.1969) (Fig. 3). Subgroup analysis based on SI scores (0 vs. 1 to 4) revealed that the higher-SI-score group had a greater percentage of university-based programs than the lower-score group (80% vs. 62%; p = 0.003). While the higher-score group was associated with a greater number of residents in the program (mean, 26.4 vs. 21.0; p = 0.04), the number of faculty (p = 0.57) and the geographic region of the program (p = 0.13) did not seem to affect the SI score (Table II). Based on multivariate analysis, both variables—the type of program (university-based vs. community-based) and the total number of residents—were independently associated with a higher SI score (p < 0.0001).

Fig. 1
Fig. 1:
Comparison of the average SI scores for university-based programs and community-based programs.
Fig. 2
Fig. 2:
A weak correlation was found between program size in terms of number of residents and the SI score (R2 = 0.074, p = 0.0004).
Fig. 3
Fig. 3:
No correlation was found between program size in terms of the number of full-time faculty and the SI score (p = 0.1969).
TABLE II - Sub-Internship (SI) Score Outcomes According to Various Factors
No Information (0) (N = 69) Some Information (1-4) (N = 82) P Value
Type of program* 0.003
 University-based 43 (62%) 66 (80%)
 Community-based 26 (38%) 16 (20%)
Size of program
 Mean no. of residents 21.0 26.4 0.04
 Mean no. of faculty 28.6 32.4 0.57
Geographic region* 0.13
 Northeast 25 (36%) 20 (24%)
 South 19 (28%) 17 (21%)
 Midwest 11 (16%) 25 (30%)
 Southwest 8 (12%) 8 (10%)
 West 6 (9%) 12 (15%)
*The values are given as the number of programs, with the percentage in parentheses.


Our study indicated that nearly half of the U.S. orthopaedic residency programs offered no information concerning a sub-internship. Previous investigators have reported on the importance of the sub-internship’s potential to generate a successful match into an orthopaedic surgery residency position3. Despite the established competitiveness of matching into orthopaedic surgery and the perceived importance of the sub-internship, the evident lack of online information concerning the orthopaedic sub-internship is concerning and warrants further attention.

A notable finding in our study was the difference in the average SI score between university-based and community-based orthopaedic surgery residency programs. This difference could be due to various reasons, such as the possibility that university-based programs keep their web sites more up to date than community-based programs do. It also could be due to the possibility that university-based programs may tend to offer sub-internships more frequently than community-based programs do and therefore could have higher corresponding SI scores. Furthermore, university-based programs may attract a different group of students than community-based programs do. The potential causes for this difference between the 2 types of programs and the possibility of self-selection require further study.

Another interesting finding was the linear relationship between the SI score and the number of residents in the program (R2 = 0.074, p = 0.0004) (Fig. 2). Additionally, there was a slightly higher total number of residents in the program web sites scoring 1 to 4 compared with those scoring 0 (mean, 26.4 vs. 21.0; p = 0.04). The size of a program may have been associated with how often the individual program updated its online information regarding the sub-internship. Again, this association may also be due to the possibility that larger programs tended to accommodate visiting students more readily than smaller programs. We found no significant difference between programs with a score of 0 and programs with a score of 1 to 4 with respect to geographic location or the number of full-time faculty.

Rozental et al.9 utilized a similar approach to ours in assessing orthopaedic surgery department online web sites and concluded that orthopaedic surgery web sites “underutilize the Internet as a source of clinical and educational services.” Our study similarly suggests that a large number of U.S. orthopaedic surgery residency programs are not fully utilizing the Internet for promoting and offering detailed information regarding sub-internship rotations to prospective resident applicants. It is plausible that programs may be able to attract more competitive applicants by having as much online information as possible concerning the sub-internship. With more information at their disposal, applicants can apply to the sub-internship with a better understanding and expectation of the program and their clinical rotation. The orthopaedic sub-internship serves as a 2-way street: program directors may use this additional exposure to assess applicants’ “fit” with the program, and applicants likewise can use this experience to assess the quality of training and culture of the program10. Thus, it is likely that programs with more detailed information about their sub-internship would give a more favorable impression to applicants than programs with little to no information about the sub-internship.

Currently, many programs utilize the Visiting Student Application Service (VSAS) in addition to the program’s web site in order to promote their sub-internship11. VSAS is a centralized application that organizes all of the orthopaedic sub-internships into 1 location and allows applicants to choose which program(s) into which they wish to enroll for a sub-internship rotation. Most students apply for elective rotations during the spring of their third year of medical school, and many do not have full access to VSAS until January or February of their third year. While VSAS provides information regarding sub-internships for prospective applicants, this information may not be complete as most programs that utilize VSAS have a link to the program web site should the applicant wish to seek additional information. Therefore, programs would still benefit from improving the information about their sub-internships on the program web site as this portal is likely the primary location that applicants browse when seeking more information about the residency program.

While our study was able to generate some interesting findings that will be relevant for both the residency applicant and the training program, it had some limitations. First, although 171 ACGME orthopaedic residency programs participated in the 2018 Match, the data-collection sheet for our study was created in 2016, when only 151 programs participated in the Match. Therefore, the additional 20 programs that participated in the 2018 Match were not included in our analysis. Additionally, there was no method to determine whether the programs that received an SI score of 0 actually offered a sub-internship but lacked online information or simply did not offer a sub-internship at all. The possibility of contacting the appropriate residency program coordinator for these programs was discussed and ultimately abandoned because of the possibility of introducing selection bias. There is also a possibility that a program may have had more in-depth information regarding a sub-internship but the information could not be found because the web page was too difficult to navigate. Additionally, while we did try to quantify the online information regarding sub-internship by using a SI score, this score was not weighted and has not been previously used or validated. Furthermore, residency program web sites are not the only way in which applicants can garner information about sub-internships. Other avenues such as counselling from the Dean of Student Affairs, mentorship from senior students, and web site forums such as Orthogate and Student Doctor Network are common methods that medical students utilize to obtain information on sub-internships that may have been overlooked in our study. Information technology and administrative support for the residency programs are other factors that can influence the availability of residency program web sites that were overlooked in our study. Future studies can assess how these factors impact the quality and availability of orthopaedic residency program web sites. Investigators also can survey matched applicants to gain further insight as to why programs are not keeping their web sites up to date and whether applicants are utilizing these web sites.

In summary, the availability and quality of online information regarding sub-internships at U.S. orthopaedic residency programs are variable. Nearly half of the programs did not have any available online information on their web sites regarding the orthopaedic surgery sub-internship. Larger and university-based orthopaedic programs seemed to have slightly more robust information regarding sub-internships than smaller and community-based programs. There needs to be greater awareness and more uniformly accessible online information regarding orthopaedic surgery sub-internship for senior medical students seeking elective orthopaedic rotations prior to applying for residency training. We believe that both student applicants and programs would benefit from an improvement in the quality of information online regarding sub-internships and that this matter needs to be further studied and discussed among educators as well as in the peer-reviewed literature.


1. The National Residency Matching Program. Results and data: 2018 Main Residency Match. 2018. Accessed 2018 Sep 19.
2. Baldwin K, Weidner Z, Ahn J, Mehta S. Are away rotations critical for a successful match in orthopaedic surgery? Clin Orthop Relat Res. 2009 Dec;467(12):3340-5. Epub 2009 Jul 7.
3. O’Donnell SW, Drolet BC, Brower JP, LaPorte D, Eberson CP. Orthopaedic surgery residency: perspectives of applicants and program directors on medical student away rotations. J Am Acad Orthop Surg. 2017 Jan;25(1):61-8.
4. Yaeger JP, Conway JH, Butteris SM, Howard CR, Moreno MA. Pediatric global health education: correlation of website information and curriculum. J Pediatr. 2013 Dec;163(6):1764-8. Epub 2013 Aug 20.
5. Ashack KA, Burton KA, Soh JM, Lanoue J, Boyd AH, Milford EE, Dunnick C, Dellavalle RP. Evaluating dermatology residency program websites. Dermatol Online J. 2016 Mar 16;22(3):13030/qt7rx3j2dn.
6. Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB. General surgery residency program websites: usefulness and usability for resident applicants. Curr Surg. 2004 Mar-Apr;61(2):236-40.
7. Skovrlj B, Silvestre J, Ibeh C, Abbatematteo JM, Mocco J. Neurosurgery residency websites: a critical evaluation. World Neurosurg. 2015 Sep;84(3):727-33. Epub 2015 May 2.
8. Svider PF, Gupta A, Johnson AP, Zuliani G, Shkoukani MA, Eloy JA, Folbe AJ. Evaluation of otolaryngology residency program websites. JAMA Otolaryngol Head Neck Surg. 2014 Oct;140(10):956-60.
9. Rozental TD, Lonner JH, Parekh SG. The Internet as a communication tool for academic orthopaedic surgery departments in the United States. J Bone Joint Surg Am. 2001 Jul;83(7):987-91.
10. Huntington WP, Haines N, Patt JC. What factors influence applicants’ rankings of orthopaedic surgery residency programs in the National Resident Matching Program? Clin Orthop Relat Res. 2014 Sep;472(9):2859-66. Epub 2014 Jun 5.
11. Association of American Medical Colleges. Visiting Student Learning Opportunities (VSLO) program. Accessed 2018 Sep 19.

Supplemental Digital Content

Copyright © 2019 The Authors. Published by The Journal of Bone and Joint Surgery, Incorporated. All rights reserved.