Secondary Logo

Journal Logo

FEATURED ARTICLES

What is the Interobserver Reliability of an Ultrasound-enhanced Physical Examination of the Hip in Infants? A Prospective Study on the Ease of Acquiring Skills to Diagnose Hip Dysplasia

Jejurikar, Neha MD1; Moscona-Mishy, León MD1; Rubio, Mónica MD2; Cavallaro, Romina MD3; Castañeda, Pablo MD1

Author Information
Clinical Orthopaedics and Related Research: September 2021 - Volume 479 - Issue 9 - p 1889-1896
doi: 10.1097/CORR.0000000000001863

Abstract

Introduction

Developmental dysplasia of the hip (DDH) is the most common disorder in newborn children [4, 11]. If diagnosed early, it can successfully be treated nonoperatively, as outlined in the most recent clinical practice guideline by the American Academy of Pediatrics [1]. The most widely used nonoperative treatment implemented in children between 0 and 6 months old is the Pavlik method [27]. However, nonoperative management with this method becomes less effective with increasing age [11, 21, 32]. Successful nonoperative management of DDH depends on early diagnosis and identification of unstable and dysplastic hips [21, 25, 27]. There is no worldwide consensus on who should undergo screening. Currently, in the United States, patients are selectively screened with ultrasound based on the presence of risk factors, specifically a family history, breech presentation, and instability of the hip on examination [7, 20]. Ultrasound has been used since the early 1980s to detect DDH during infancy, effectively decreasing the incidence of late diagnosis [5, 9, 10, 12, 30]. This can have a dramatic effect on the lives of patients who are otherwise subjected to operative treatment. Ultrasound examinations are commonly performed by a radiology technician rather than a clinician. The downside to using ultrasound for diagnosis is that in inexperienced hands, the results of an ultrasound examination can be misleading. Adequate training and practice empower clinicians with the tool of ultrasound for diagnostic purposes.

The rationale behind this study was to understand whether clinicians can be trained efficiently and effectively to diagnose dysplasia using ultrasound-enhanced physical examinations. We aimed to fill the void in research examining the effectiveness of ultrasound training in orthopaedic education. Although ultrasound training is effective in obstetrics and emergency medicine training, there is a relative lack of research related to orthopaedic education [2, 3, 13]. Although some studies have looked at interobserver reliability of ultrasound evaluation of the infant hip [14, 22, 24], as well as limitations of the technique [17, 31], we are not aware of any study on the effectiveness of ultrasound training in detecting DDH. Addressing concerns about the user reliability and proficiency of ultrasound may inform the current lack of dedicated research and hesitancy to use ultrasound among clinicians diagnosing DDH.

We therefore asked, can ultrasound be taught to and used reliably by different providers to identify DDH in neonates?

Patients and Methods

Study Design and Setting

This prospective trial was designed to assess whether ultrasound-enhanced physical examinations can be taught to clinicians in order to diagnose DDH. The primary endpoint was the comparison of the three learners’ ultrasound results against those of a clinician who is experienced in using ultrasound-enhanced physical examinations. The study was conducted over the course of 4 months in an outpatient setting in the practice of a single surgeon. The cohort was composed of a series of patients younger than 6 months presenting to a specialty pediatric orthopaedic practice for a hip evaluation because they had a positive or equivocal physical examination result or were at a high risk of having DDH.

Patients

Of the 227 infants, 186 were girls and 41 were boys. The mean age of the studied patients was 7 weeks (range 1 to 23 weeks, or 6 to 162 days).

Of the 227 infants, 9% (21 of 227) were referred by a pediatrician because of an abnormal examination result, with 16 being Ortolani-positive and five being Barlow-positive. A total of 42% (95 of 227) were referred because of an equivocal examination result consisting of a hip click, asymmetric gluteal or thigh folds, perceived limb length discrepancy, or limited hip abduction. A total of 111 patients were referred because of a substantial risk factor, with 19% (21 of 111) of these patients being breech at the time of delivery (Table 1). The other risk factors that triggered a referral were a family history of DDH in a first-degree relative in 16% (18 of 111) of patients and a combination of either being the first-born child or being a girl in the remainder. Infants with known genetic abnormalities were excluded.

Table 1. - Reason for a referral from a pediatrician for an evaluation for DDH
Reason for referral from pediatrician % (n) of infants
(n = 227)
Abnormal examination findings 9 (21)
 Ortolani-positive 7 (16)
 Barlow-positive 2 (5)
Equivocal examination findings: hip click, asymmetric gluteal or thigh folds, perceived limb length discrepancy, limited hip abduction 42 (95)
Major risk factor 49 (111)
 Breech at the time of delivery 9 (21)
 Family history or a first-degree relative with DDH 8 (18)
 First-born child and/or girl 32 (72)

Training Program

The 2-hour training course consisted of a standardized course of three video-based learning modules that the learners would watch after beginning their training. The first video explained the development of ultrasound-enhanced physical examinations of the hip (Supplemental Video, https://journals.lww.com/clinorthop/pages/videogallery.aspx) the second showed examples of the types of outcomes considered for the examination, and the third reinforced the concepts explained in the first two videos. Each video was 6 minutes long and was provided to the learners to continue to watch on their own time. After watching the video-based lectures, the learners were able to ask questions and receive clarification from the gold-standard examiner. The learners then observed and were guided through an examination, watching the gold-standard examiner perform the technique twice and then performing it under direct supervision once, with the teacher helping guide the position and direction of the probe as well as the ideal capture of images. For each learner, this observational and hands-on experience lasted 90 minutes and included three patients (six hips).

Study Participants and Evaluations

After the training of the three learners as described above, each patient underwent an examination by two physicians. One (PC) was an experienced clinician and was considered the gold-standard examiner for this study. The gold-standard examiner was an experienced pediatric orthopaedic surgeon who had been in practice for 16 years after fellowship and has a dedicated pediatric hip practice. The other examiner was one of three different providers: a pediatric orthopaedic fellowship–trained surgeon (MR) with 4 years of experience, a single fifth-year orthopaedic surgery resident (RC), and a single board-certified pediatrician (who was not an author on this paper) with 3 years of clinical experience. All of the second examiners were trained by the senior examiner in a 2-hour one-on-one training session as described above. Only these four examiners performed the examinations.

The technique of ultrasound-enhanced physical examination of the hip allows one of four possible outcomes: normal, dysplastic, unstable, and dislocated (Fig. 1). Normal implies that the femoral head is in contact with the acetabulum, has less than 4 mm of excursion on dynamic stress testing, and has a deep acetabular shape (with at least 50% bony coverage and an alpha angle greater than 60°). Dysplastic hips are those that are located and stable under stress testing but have a shallow acetabulum, unstable hips have greater than 4 mm of excursion under stress testing, and dislocated hips are those in which the femoral head is not in contact with the acetabulum in a resting state (Fig. 2). The outcome can also be reported as either a normal or abnormal outcome (Fig. 3). All three pathologic categories (dysplasia, instability, and dislocation) are considered abnormal.

F1
Fig. 1:
This radiograph shows the pelvis of an infant.
F2
Fig. 2:
This radiograph shows the pelvis of a newborn child, overlaid by ultrasound images of the hip demonstrating a located hip on the right and a dislocated hip on the left.
F3
Fig. 3:
Alpha and beta angles were measured using ultrasonography to assess for hip dysplasia.

A total of 227 infants (454 hips) underwent an ultrasound-enhanced physical examination of the hip by two different examiners. This was performed using point-of-care ultrasound of the infants’ hips using a portable ultrasound device that was taken into the examination room for each patient. The device is made of anodized aluminum incorporating ultrasound-on-a-chip technology, wherein 9000 capacitive micromachined ultrasound transducers create and receive sound from 1 to 10 MHz, which works by sending an electric current through a crystal composite (lead zirconate titanate). This material acts as a transducer, converting electrical energy into sound waves that bounce off structures inside the body. On their return to the crystal transducer, these echoes are turned back into electrical signals and processed by a computer into a two-dimensional moving image.

Each patient underwent two examinations in one day, independent of each other. Each examiner performed an ultrasound-enhanced physical examination of the hip on their own. The trainees performed their examinations first and the gold-standard examiner performed his examinations second. Each examiner then analyzed their patient-specific results on two separate occasions: one at the time of image acquisition and a second when all data had been collected in order to determine the intraobserver reliability. The second analysis was undertaken on each examiner’s time. All data were collected on a standardized spreadsheet, on which the patient record and outcome were documented.

The results were then analyzed by a fifth independent blinded reviewer (LMM), who was familiar with the technique, to determine agreement among the examiners.

Reliability is a measure of the extent to which observations can be replicated and whether a measurement is the same after repeated trials. Interrater reliability is the extent of agreement among raters scoring the same participants under the same conditions. Intrarater reliability is the extent of agreement between the same raters evaluating the same participants a second time.

Primary Study Outcome

Our primary study goal was to assess the effectiveness of training clinicians in the use of ultrasound-enhanced physical examinations in order to diagnose DDH. To achieve this, we studied the reliability of the ultrasound-enhanced examinations performed by three learners and an experienced clinician.

Ethical Approval

Ethical approval for this study was obtained from the New York University School of Medicine (study number i17-0096_MOD09). The parents of the included patients were informed of the study, and written consent was obtained.

Statistical Analysis

Interrater reliability was determined from one set of measurements obtained by two observers. Intrarater reliability was determined from a second evaluation of the same set of measurements made by a single observer. Both reliability parameters were analyzed in the same way with the intraclass correlation coefficient (ICC) using a mixed model analysis. The statistical analysis was performed using SAS software.

All ICCs were interpreted using the Rosner [26] criteria: 0 to 0.40 = poor agreement, > 0.40 to 0.75 = good agreement, and > 0.75 to 1.00 = excellent agreement. The higher the ICC, the stronger the correlation. A high correlation means the measurement can be performed reliably by different observers or can be performed reliably by the same person multiple times [8]. A result of 1 represents perfect agreement and 0 represents no agreement [16]. Coverage probability was determined by using the frequentist method of the generalized CI, set nominally at 95%.

For the 16 hips that were referred for being Ortolani-positive, the gold-standard examiner found that 12 were dislocated at rest and were reducible. Of the other four hips, one was dislocated but irreducible, one was unstable, and two were normal. Five hips were dislocated at rest: two in the group that was referred because of a hip click, one from the group that was considered Barlow-positive, and two from the group considered at risk (Table 2).

Table 2. - Reason for referral and ultimate outcome according to the gold-standard examinera
Parameter % (n) Ortolani-positive
(n = 16)
% (n) Barlow-positive (n = 5) % (n) breech at birth (n = 42) % (n) first-degree relative with hip dysplasia
(n = 36)
Dislocated and reducible 75 (12) 20 (1) 5 (2) 3 (1)
Dislocated and irreducible 6 (1) 0 (0) 0 (0) 0 (0)
Unstable dysplasia 6 (1) 40 (2) 7 (3) 0 (0)
Stable dysplasia 0 (0) 0 (0) 19 (8) 6 (2)
Normal 13 (2) 40 (2) 69 (29) 92 (33)
aOrtolani-positive and Barlow-postive data are presented as number of patients, whereas breech at birth and first-degree relative with hip dysplasia are presented as number of hips.

Among the five hips referred for being Barlow-positive, the gold-standard examiner found that two were unstable under stress, one was dislocated and reducible, and two were normal. Another 22 hips were unstable on ultrasound: one from the clinically Ortolani-positive group, two from the group with a click, and 19 from the group considered at risk.

Of the 21 patients referred for having been breech at the time of birth, the gold-standard examiner found that two had a dislocation (one was unilateral and one had a dislocation on one side and stable dysplasia on the other); three had unilateral, unstable hips; three had unilateral, stable dysplasia; and two had bilateral, stable dysplasia.

Of the 18 patients referred for having a first-degree relative with hip dysplasia, only one had unilateral dislocation and two had stable dysplasia, according to the gold-standard examiner.

Of the 227 patients (454 hips), there were 18 dislocations, 24 unstable hips, 63 dysplastic hips, and 349 normal hips (as graded by the gold-standard examiner). This was determined at the first visit, and the gold-standard examiner was the second person to perform the test.

On a secondary review of the obtained images, the gold-standard examiner found the same 18 dislocations and 24 unstable hips, but subsequently identified one extra hip as being dysplastic, bringing the total number of hips with dysplasia to 64.

Intraobserver Reliability for Learners

Following the described program of instruction, the intraobserver reliability for each learner was generally very high (Table 3). The intraobserver agreement for Examiner 1 was 0.995 for dislocation, 0.923 for instability, and 0.913 for dysplasia. The overall intraobserver agreement for Examiner 1 was 0.928. The intraobserver agreement for Examiner 2 was 1.00 for dislocation, 0.916 for instability, and 0.891 for dysplasia. The overall intraobserver agreement for Examiner 2 was 0.923. The intraobserver agreement for Examiner 3 was 1.00 for dislocation, 0.901 for instability, and 0.926 for dysplasia. The overall intraobserver agreement for Examiner 3 was 0.932. The intraobserver agreement for the gold-standard examiner was 1.00 for dislocation and instability and 0.99 for dysplasia. The overall intraobserver agreement for the gold-standard examiner was 0.999 (Table 3).

Table 3. - ICCs for intraobserver reliability using a spectrum of possible outcomes (the same examiner reviewing their own images a second time)
Parameter Gold-standard examiner Examiner 1 Examiner 2 Examiner 3
Dislocation 1.00 0.995 1.00 1.00
Instability 1.00 0.923 0.916 0.901
Dysplasia 0.999 0.913 0.891 0.926
Normal 0.999 0.881 0.897 0.903
Mean 0.999 0.928 0.923 0.932

When using only the binary outcome of normal or abnormal, we found the intraobserver agreement was 0.999 for the gold-standard examiner, 0.991 for Examiner 1, 0.989 for Examiner 2, and 0.998 for Examiner 3. The mean intraclass and intraobserver agreement for all examiners was 0.994 (Table 4).

Table 4. - ICCs for intraobserver reliability using a binary outcome (the same examiner reviewing their own images a second time)
Parameter Gold standard Examiner 1 Examiner 2 Examiner 3
Abnormal 0.999 0.991 0.989 0.998
Normal

Results

Agreement Between Learners and Experienced Ultrasonographer

Our study demonstrated that ultrasound can be taught to physicians in order to reliably diagnose DDH in neonates. After a 2-hour course, the three learners had very high agreement with the experienced examiner for diagnosing DDH.

Examiner 1 identified 17 dislocations, 21 unstable hips, and 73 dysplastic hips at the initial examination. A subsequent analysis of images yielded 18 dislocations, 23 unstable hips, and 78 dysplastic hips. One of the hips initially considered dislocated was subsequently deemed to be unstable in Examiner 1’s experience.

Examiner 2 identified 18 dislocations, 23 unstable hips, and 64 dysplastic hips at the initial examination. A subsequent analysis of images yielded 18 dislocations, 20 unstable hips, and 70 dysplastic hips.

Examiner 3 identified 18 dislocations, 25 unstable hips, and 72 dysplastic hips at the initial examination. A subsequent analysis of images yielded 18 dislocations, 24 unstable hips, and 73 dysplastic hips.

When comparing the outcome as spectrum of pathologic findings (normal, dysplasia, instability, and dislocation), the interobserver agreement between the gold-standard examiner and Examiner 1 was 0.942, between the gold-standard examiner and Examiner 2 was 0.904, and between the gold-standard examiner and Examiner 3 was 0.923. The ICC between the gold-standard examiner and the other examiners for all hips was 0.915 (p = 0.001) (Table 5).

Table 5. - ICCs for interobserver reliability for a spectrum of possible outcomes (agreement between independent examiners and gold standard)
Parameter Examiner 1 Examiner 2 Examiner 3
Dislocation 1.00 0.971 1.00
Instability 0.909 0.901 0.902
Dysplasia 0.901 0.854 0.902
Normal 0.876 0.885 0.891
Mean 0.942 0.904 0.923
95% CI 0.907-0.981 0.873-0.928 0.881-0.956

When comparing the outcome as a binary outcome (normal or abnormal), we found the interobserver agreement between the gold-standard examiner and Examiner 1 was 1.00, between the gold-standard examiner and Examiner 2 was 0.960, and between the gold-standard examiner and Examiner 3 was 0.961 (Table 6). The ICC between the gold-standard examiner and the other examiners for all hips and the binary outcome was 0.973 (p = 0.001) (Table 7).

Table 6. - ICCs for interobserver reliability using a binary outcome (agreement between independent examiners and gold-standard examiner)
Parameter Examiner 1 Examiner 2 Examiner 3
Abnormal 1.00 0.970 0.991
Normal 1.00 0.951 0.932
Mean 1.00 0.960 0.961
95% CI 0.982-1.00 0.949-0.981 0.951-0.989

Table 7. - ICCs for both intraobserver and interobserver reliability using both a spectrum and binary outcomes
Parameter Intraobserver Interobserver
Spectrum 0.945 (95% CI 0.914-0.965) 0.915 (95% CI 0.884-0.955)
Binary 0.994 (95% CI 0.974-0.998) 0.973 (95% CI 0.963-0.989)

Discussion

DDH is the most common disorder in newborns, and operative management can sometimes be prevented by early diagnosis. This study’s aim was to assess the effectiveness of a short training program in teaching physicians how to conduct an ultrasound-enhanced physical examination in newborn babies. Additionally, we aimed to assess the reliability of examination findings among clinicians, given the poor reliability of physical examination alone shown by the high variability in findings by referring physicians. We found that an ultrasound-enhanced physical examination of the hips was easy to learn, and diagnoses were reliable among the physicians studied.

Limitations

This study has some limitations. First, although the number of patients tested was high, the number of physicians evaluated in the training program was limited. The aim of the study was twofold: to assess the effectiveness of an ultrasound training program and to compare reliability among clinicians. To enable each clinician to examine a high number of patients, the number was kept low for this study. However, a future similar study of larger numbers of trained clinicians would be beneficial.

Second, the assessment of acquisition of skills was tested shortly after the training session. Long-term retention was not a part of this study design. Our aim was to determine whether the diagnosis could be effectively taught. We successfully achieved this aim. Future studies might test the long-term retention of skills.

Third, all clinicians were taught and subsequently compared with a single experienced clinician. This was done to maintain clarity for a gold-standard diagnosis. There would be benefit in assessing the teaching capabilities of multiple providers. The instructional videos we used have been made available for teaching purposes.

Finally, all patients were examined at a single pediatric orthopaedic practice, and there is likely selection bias, because most referred patients had some degree of pathologic findings. Although the study had a large patient population, a multicenter trial would demonstrate better generalizability. The patient population was characteristic of patients referred to a pediatric orthopaedist for an evaluation of hip dysplasia. Although one surgeon was included in this study, pediatric orthopaedic surgeons at academic centers would have similar patient populations for teaching purposes. There is a potential for assessment or unblinded outcome bias because the second examiner could have been made aware of the findings of the first. However, given that the second examiner was the gold standard for every patient, there is very little risk that this influenced the results.

Agreement Between Learners and Experienced Ultrasonographer

A short (2-hour) intervention resulted in very high agreement between learners and one experienced ultrasonographer in diagnosing DDH. This is important because it demonstrates physician learners can effectively acquire skills to independently diagnose DDH using ultrasound. The ease of using point-of-care ultrasound can make the diagnosis of DDH simple for both pediatricians and orthopaedists in the office setting. Implementation and focus on ultrasound training in pediatric and orthopaedic residency programs can have implications for the widespread use of ultrasound for diagnosis among clinicians in the future. Our study is novel because it describes a training method that is easier to teach than other studied methods. Karakus et al. [14] showed no agreement in findings for beta angles between residents, in addition to a lack of agreement among residents and pediatric orthopaedic surgeons (Fig. 2). A key difference in our training method is the use of binary outcomes (identification of normal versus abnormal) as opposed to reliance on numerical angles. The ease of classifying dysplasia based on binary outcomes compared with alpha and beta angles has already been demonstrated by Simon et al. [29]. Mostofi et al. [18] showed that novice operators had poor reliability with two-dimensional ultrasound; however, they demonstrated high reliability with three-dimensional ultrasound. Our study demonstrates that a training program can have a positive impact on reliable and accurate ultrasound techniques in novice operators. Similar to other studies on ultrasound training [28, 33], our study shows that a training course improves the reproducibility of learners. Considering the importance of early introduction of skills acquisition [15] and the success of simulator training programs for other skills [2, 6, 23], orthopaedic and pediatric residency programs would benefit from implementing ultrasound training for diagnosing DDH.

Conclusion

Although the methods described by Graf [9, 10] and by Harcke and Grissom [12] and Clarke [5] have proven value [19, 29, 30, 34], the difficulty with training and the uniform implementation of standardized programs has limited their effectiveness. The objective of the current study was to provide a simpler method that can be taught and implemented with ease. With the incorporation of advanced practice providers in many medical settings, a simple and easy method to improve the early diagnosis of DDH is paramount, and future directions of research should include non-physician providers, which will ultimately make the technique more widely available. Another encouraging future direction is incorporating a binary outcome into artificial intelligence software that would allow the identification of pathologic findings regardless of the user. Our results show that this is within the realm of reality.

References

1. American Academy of Pediatrics. Clinical practice guideline: early detection of developmental dysplasia of the hip. Committee on Quality Improvement, Subcommittee on Developmental Dysplasia of the Hip. Pediatrics. 2000;105:896-905.
2. Abuhamad A, Minton KK, Benson CB, et al. Obstetric and gynecologic ultrasound curriculum and competency assessment in residency training programs: consensus report. J Ultrasound Med. 2018;37:19-50.
3. Alrahmani L, Codsi E, Borowski KS. The current state of ultrasound training in obstetrics and gynecology residency programs: ultrasound training in OB-GYN residency programs. J Ultrasound Med. 2018;37:2201-2207.
4. Benson M, Wainwright A. Developmental dysplasia of the hip. Surgery. 2004;22:5-9.
5. Clarke NM, Harcke HT, McHugh P, Lee MS, Borns PF, MacEwen GD. Real-time ultrasound in the diagnosis of congenital dislocation and dysplasia of the hip. J Bone Joint Surg Br. 1985;67:406-412.
6. Cychosz CC, Tofte JN, Johnson A, Gao Y, Phisitkul P. Fundamentals of arthroscopic surgery training program improve knee arthroscopy simulator performance in arthroscopic trainees. Arthroscopy. 2018;34:1543-1549.
7. Dogruel H, Atalar H, Yavuz OY, Sayli U. Clinical examination versus ultrasonography in detecting developmental dysplasia of the hip. Int Orthop. 2008;32:415-419.
8. Eriksson P, Kovalainen A. Qualitative Methods in Business Research. 2nd ed. Sage; 2015.
9. Graf R. New possibilities for the diagnosis of congenital hip joint dislocation by ultrasonography. J Pediatr Orthop. 1983;3:354-359.
10. Graf R. Fundamentals of sonographic diagnosis of infant hip dysplasia. J Pediatr Orthop. 1984;4:735-740.
11. Guille JT, Pizzutillo PD, MacEwen GD. Development dysplasia of the hip from birth to six months. J Am Acad Orthop Surg. 2000;8:232-242.
12. Harcke HT, Grissom LE. Infant hip sonography: current concepts. Semin Ultrasound CT MR. 1994;15:256-263.
13. Hempel D, Schröper T, Pfister R, Michels G. Ultrasound training in emergency and intensive care medicine: integration already in medical school? [in German]. Med Klin Intensivmed Notfmed. 2019;114:519-524.
14. Karakus O, Karaman O, Sari AS, Orak MM, Muratli HH. Is it difficult to obtain inter-observer agreement in the measurement of the beta angle in ultrasound evaluation of the paediatric hip? J Orthop Surg Res. 2019;14:221.
15. Laird-Fick HS, Chang C, Wang L, et al. Assessing the growth in clinical skills using a progress clinical skills examination. PeerJ. 2020;8:e9091.
16. Li Gwet K. Handbook of Inter‐Rater Reliability. 4th ed. Advanced Analytics LLC; 2014.
17. Marks DS, Clegg J, al-Chalabi AN. Routine ultrasound screening for neonatal hip instability. Can it abolish late-presenting congenital dislocation of the hip? J Bone Joint Surg Br. 1994;76:534-538.
18. Mostofi E, Chahal B, Zonoobi D, et al. Reliability of 2D and 3D ultrasound for infant hip dysplasia in the hands of novice users. Eur Radiol. 2019;29:1489-1495.
19. Omeroğlu H. Use of ultrasonography in developmental dysplasia of the hip. J Child Orthop. 2014;8:105-113.
20. Omeroğlu H, Koparal S. The role of clinical examination and risk factors in the diagnosis of developmental dysplasia of the hip: a prospective study in 188 referred young infants. Arch Orthop Trauma Surg. 2001;121:7-11.
21. Ömeroğlu H, Köse N, Akceylan A. Success of Pavlik harness treatment decreases in patients ≥ 4 months and in ultrasonographically dislocated hips in developmental dysplasia of the hip. Clin Orthop Relat Res. 2016;474:1146-1152.
22. Orak MM, Onay T, Çağırmaz T, Elibol C, Elibol FD, Centel T. The reliability of ultrasonography in developmental dysplasia of the hip: how reliable is it in different hands? Indian J Orthop. 2015;49:610-614.
23. Reppenhagen S, Weißenberger M, Barthel T, Rudert M, Anetzberger H. Virtual arthroscopy: gaming or training concept of the future [in German]. Unfallchirurg. 2019;122:431-438.
24. Roposch A, Graf R, Wright JG. Determining the reliability of the Graf classification for hip dysplasia. Clin Orthop Relat Res. 2006;447:119-124.
25. Rosenberg N, Bialik V, Norman D, Blazer S. The importance of combined clinical and sonographic examination of instability of the neonatal hip. Int Orthop. 1998;22:185-188.
26. Rosner BA. Fundamentals of Biostatistics. 7th ed. Brooks/Cole Cengage Learning; 2010.
27. Samora J, Quinn RH, Murray J, Pezold R, Hall Q. Management of developmental dysplasia of the hip in infants up to six months of age: intended for use by orthopaedic specialists. J Am Acad Orthop Surg. 2019;27:e360-e363.
28. Shirai Y, Wakabayashi K, Wada I, et al. Reproducibility of acquiring ultrasonographic infant hip images by the Graf method after an infant hip ultrasound training course. J Med Ultrason (2001). 2018;45:583-589.
29. Simon EA, Saur F, Buerge M, Glaab R, Roos M, Kohler G. Inter-observer agreement of ultrasonographic measurement of alpha and beta angles and the final type classification based on the Graf method. Swiss Med Wkly. 2004;134:671-677.
30. Terjesen T, Rundén TO, Tangerud A. Ultrasonography and radiography of the hip in infants. Acta Orthop Scand. 1989;60:651-660.
31. Tönnis D, Storch K, Ulbrich H. Results of newborn screening for CDH with and without sonography and correlation of risk factors. J Pediatr Orthop. 1990;10:145-152.
32. Upasani VV, Bomar JD, Matheney TH, et al. Evaluation of brace treatment for infant hip dislocation in a prospective cohort: defining the success rate and variables associated with failure. J Bone Joint Surg Am. 2016;98:1215-1221.
33. Vathana T, Rust S, Mills J, et al. Intraobserver and interobserver reliability of two ultrasound measures of humeral head position in infants with neonatal brachial plexus palsy. J Bone Joint Surg Am. 2007;89:1710-1715.
34. Wientroub S, Grill F. Ultrasonography in developmental dysplasia of the hip. J Bone Joint Surg Am. 2000;82:1004-1018.
© 2021 by the Association of Bone and Joint Surgeons