Journal Logo

Research Article

Smartphone Data Capture Efficiently Augments Dictation for Knee Arthroscopic Surgery

Featherall, Joseph BS; Oak, Sameer R. MD; Strnad, Gregory J. MS; Farrow, Lutul D. MD; Jones, Morgan H. MD, MPH; Miniaci, Anthony A. MD, FRCSC; Parker, Richard D. MD; Rosneck, James T. MD; Saluan, Paul M. MD; Spindler, Kurt P. MD

Author Information
Journal of the American Academy of Orthopaedic Surgeons: February 1, 2020 - Volume 28 - Issue 3 - p e115-e124
doi: 10.5435/JAAOS-D-19-00074
  • Open

Abstract

As health care becomes increasingly value driven, the ability to justify treatment is enhanced by prospective, high-quality, standardized databases.1 Examples of large anterior cruciate ligament reconstruction (ACLR) databases include the Swedish National Anterior Cruciate Ligament (ACL) Register (SNKRL), Danish Cruciate Ligament Registry (DKKR), Kaiser Permanente Anterior Cruciate Ligament Reconstruction Registry (KPACLRR), and the Multicenter Orthopaedic Outcomes Network and Multicenter ACL Revision Study cohort.2-6 The uses of high-quality surgical data include, but are not limited to, internal quality improvement initiatives, large-scale comparative effectiveness research, outcomes research, cost-effectiveness research, and clinical trials.7-9 Such data have greatly contributed to our understanding of surgical practice.10 The availability of prospective, standardized, high-quality data is a foundational component for the advancement of the field of orthopaedic surgery.

Several practical challenges exist in constructing large high-quality data sets. Much of the important data that are captured in the electronic medical record (EMR) cannot be accessed easily for statistical analysis.11 Error rates in the manual extraction of data from EMRs have been reported from 8% to 23%, with data varying by site, clinical area, and surgical specialty.12 Increasing the accuracy of such data through quality assurance and data review increases cost.12 Operative reports, a key data source for surgical research, tend to infrequently report quantitative data, markedly limiting the precision of research and quality measurement.13 Even best-in-class registry data are collected through a secured web-based or paper form, which have obvious limitations in workflow (both), data accessibility (paper), and an exhaustivity of 85% to 90% for surgeon-reported data.10

To further advance orthopaedic outcomes measurement nationally, investigators developed the OrthoMiDaS (Orthopaedic Minimal Data Set) Episode of Care (OME) database and data collection methodology. OME's goal is to accurately and consistently collect patient-reported outcome measures (PROMs) immediately before and at the time of peak function after high-volume elective orthopaedic surgeries, as well as to accurately and consistently collect information about the actual surgical intervention in a manner that is both faster and more detailed than previously designed outcomes database systems.

This study describes the development of the OME database and provider-friendly, evidence-based smartphone data collection methodology and assesses the performance of the OME data capture system on the dimensions of agreement with operative note and implant log, consistency of data, and speed of provider input. We hypothesized that the use of the unique, provider-friendly, smartphone-based OME data capture system would increase the quality of orthopaedic procedure data in the context of ACLR and meniscal repair arthroscopic procedures.

Methods

OrthoMiDaS Episode of Care Database Design

A multidisciplinary team including administrators, orthopaedic surgeons, and software developers was assembled for design purposes. The OME database collects the following two distinct classes of data: PROMs and procedural data about the orthopaedic surgery itself. PROMs collected for arthroscopic knee surgeries include the Knee injury and Osteoarthritis Score, The Hospital for Special Surgery Pediatric Functional Activity Brief Scale, and the Veterans Rand 12-Item Health Survey.14-17 Upon check-in at the surgery center, patients receive an iPad with their PROM form and complete it before going back into the preoperative holding area. This process is designed to be built into the standard clinical workflow so as not to slow down the operating rooms or require any additional staffing to execute.

The second class of data, the procedural details about surgery, are entered into OME by the surgeons themselves immediately after surgery using their hospital-issued smartphones or desktop/laptop computers (Figure 1). Demographics such as height, weight examination under anesthesia findings, commonly cited operative report parameters, and key predictors of surgical outcomes for ACLR, meniscal repair, and cartilage repair as identified in the literature are collected in the form of discrete data entry fields.6,18-20 In total, OME currently provides 449 fields in the knee arthroscopy surgery data set. Branching logic is used to streamline provider data input. For primary meniscal repair and ACLR procedures, the branching logic system will require the surgeon to input approximately 13 fields for each meniscal repair and approximately 16 fields for the ACLR. However, the number of fields will expand as complexity increases. The basic set of meniscal pathology and repair data are shown in Appendix 1. Within the user interface, the branching logic displays fields only relevant to the individual procedure and pathology, greatly expediting the data collection process and decreasing the cognitive load on the surgeon. For example, selecting meniscal repair presents the surgeon with repair types, followed by manufacturers of implants/devices, and then specific device/implant offerings by that company, guiding the surgeon to procedure-specific details rather than requiring unnecessary data input as is often the case in EMR template usage. On the morning of surgery, surgeons receive reminder e-mails for each surgery they are about to perform that day with links to the surgeries' corresponding OME surgeon forms; like the PROMs-collection workflow, surgical data collection is built into surgeons' workflows to minimize delays in the daily routine.

Figure 1
Figure 1:
OrthoMiDaS Episode of Care iPhone interface demonstrating detailed data capture and speed dial provider–specific templates. ACL = anterior cruciate ligament, ACLR = anterior cruciate ligament reconstruction

Architecturally, OME exists as a collection of research electronic data capture (REDCap) databases managed by the custom-built software that manages the multiplatform data entry and distribution.21,22 All software is hosted locally and was approved by the local institutional review board (IRB) and information security. This study was also approved by the local IRB (IRB# 06-196).

Patient Selection

The OME database was launched on February 18, 2015. One hundred patients undergoing ACLR, meniscal repair, partial meniscectomy, or a combination of these procedures were included in the data set from the first four months of data collection (February through June 2015). All 12 surgeons at the sports health center who performed these procedures were included in the data set.

Data Collection and Validation

OME data used in this study were prospectively captured by the surgeons directly following their surgical procedures and were exported into a study database in an automated fashion. To evaluate the agreement with surgeons' operative dictation and/or implant logs, an independent REDCap database was established. Independent chart review data for comparison were collected from the operative report and the implant log in the Epic EMR system (Epic Systems). Reviewers of the operative report and the implant log were blinded to the OME REDCap results. Before analysis, the two data sets were reviewed for discrepancies, and all unmatching data were rechecked.

Assumptions in Chart Review Data Collection

The following assumptions were made in obtaining chart-reviewed operative dictation data: (1) If only one tunnel or reamer size was specified, both tibial and femoral tunnels were assumed to be the same size. (2) The operative report was used to determine the implant number and type; when a discrepancy or lack of clarity arose, the reviewer deferred to the implant log. (3) If the bone reamer or tunnel sizes were not specified, the graft size was used to approximate the tunnel size. (4) Regarding ACL status, normal status and no status were considered equivalent.

Statistical Analyses

All analyses were performed using R software (R version 3.2.3 [2015-12-10]). Agreement on nominal variables was measured by Cohen's (unweighted) kappa. 0.81 to 1.00 was considered almost perfect agreement; 0.61 to 0.80 was considered substantial agreement; and 0.41 to 0.60 was considered moderate agreement.23,24 Agreement on numeric variables was measured by the concordance correlation coefficient (CCC). 0.99 to 1.00 was considered almost perfect agreement; 0.95 to 0.99 was considered substantial agreement; 0.90 to 0.95 was considered moderate agreement; and <0.90 was considered poor agreement.25 Operative report and OME completion rates are presented as percent completion and are compared using the McNemar test (with continuity correction).

Results

Basic Demographics

The sample included 100 patients undergoing arthroscopic ACLR, arthroscopic meniscal repair, or both. The median age of the patients was 23 years (95% confidence interval [CI] [24.2, 28.7]), and 48% (n = 48) were women. This sample included 94 ACLRs, 58 partial meniscectomies, and 26 meniscal repairs as described in Table 1.

Table 1 - Knee Arthroscopy Patient Characteristics
Factor N (%) N
Characteristic
 N 100
 Female 48 (48.0%)
 Median age (95% CI) 23 (24.2, 28.7)
 Operative limb 100
  Both 1 (1.00%)
  Left 45 (45.0%)
  Right 54 (54.0%)
Medial meniscus
 Meniscus status 100
  Complete tear 28 (28.0%)
  Normal 55 (55.0%)
  Partial tear 17 (17.0%)
 Main tear type 45
  Bucket-handle 9 (20.0%)
  Complex 7 (15.6%)
  Horizontal 2 (4.44%)
  Longitudinal 27 (60.0%)
 Location to horns 45
  Both 8 (17.8%)
  Posterior only 37 (82.2%)
 Location to blood supply 36
  Red-red 14 (38.9%)
  Red-white 16 (44.4%)
  White-white 6 (16.7%)
 Main tear length 45
  <6 mm 5 (11.1%)
  6–10 mm 4 (8.89%)
  11–15 mm 11 (24.4%)
  16–20 mm 18 (40.0%)
  >20 mm 7 (15.6%)
 Treatment 45
  Abrade + trephine 1 (2.22%)
  No treatment for tear 9 (20.0%)
  Partial excision 15 (33.3%)
  Repair 20 (44.4%)
 Repair technique 20
  All inside 19 (95.0%)
  Both inside-out and all-in 1 (5.00%)
 Implant manufacturer 20
  Arthrex 3 (15.0%)
  Smith & Nephew 17 (85.0%)
 Implant system 20
  DartStick 3 (15.0%)
  FAST-FIX 360 17 (85.0%)
 Number of implants Med [Q1, Q3] 2.00 [2.00; 3.00] 20
Lateral meniscus
 Meniscus status 100
  Complete tear 48 (48.0%)
  Normal 43 (43.0%)
  Partial tear 9 (9.00%)
 Main tear type 57
  Bucket-handle 5 (8.77%)
  Complex 12 (21.1%)
  Horizontal 3 (5.26%)
  Longitudinal 12 (21.1%)
  Oblique/flap 9 (15.8%)
  Radial 14 (24.6%)
  Root 2 (3.51%)
 Location to horns 57
  Anterior only 4 (7.02%)
  Both 13 (22.8%)
  Posterior only 40 (70.2%)
 Location to blood supply 17
  Red-red 3 (17.6%)
  Red-white 7 (41.2%)
  White-white 7 (41.2%)
 Main tear extent 16
  Complete to periphery 4 (25.0%)
  Partial, periphery intact 12 (75.0%)
 Main tear length 57
  <6 mm 10 (17.5%)
  6–10 mm 14 (24.6%)
  >20 mm 11 (19.3%)
  11–15 mm 19 (33.3%)
  16–20 mm 3 (5.26%)
  >20 mm 11 (19.3%)
 Treatment 57
  No treatment for tear 8 (14.0%)
  Partial excision 43 (75.4%)
  Repair 6 (10.5%)
 Repair technique 6
  All inside 5 (83.3%)
  Inside-out 1 (16.7%)
 Implant system 5
  DartStick 1 (20.0%)
  FAST-FIX 360 4 (80.0%)
 Number of implants, Med [Q1, Q3] 3.00 [2.00; 4.00] 5
Anterior cruciate ligament
 ACL status 100
  Complete tear 91 (91.0%)
  Normal 6 (6.00%)
  Partial tear 3 (3.00%)
 ACLR performed 94
  Primary 93 (98.9%)
  Revision 1 (1.06%)
 Graft 94
  ALLO Achilles 5 (5.32%)
  ALLO other (HG, TA, PT, and ITB) 1 (1.06%)
  Auto BTB 32 (34.0%)
  Auto HG 56 (59.6%)
 Number of strands, Med [Q1, Q3] 4.00 [1.00; 5.00] 94
 Femur tunnel (mm), Med [Q1, Q3] 9.00 [8.50; 10.0] 94
 Tibia tunnel (mm), Med [Q1, Q3] 9.00 [8.62; 10.0] 94
 Primary femoral fixation 94
  Cross-pin 8 (8.51%)
  Interference screw 32 (34.0%)
  Suspensory 54 (57.4%)
 Primary femoral screw type 32
  Bioabsorbable 14 (43.8%)
  Metal 18 (56.2%)
 Primary tibial fixation 94
  Interference screw 77 (81.9%)
  Suspensory 17 (18.1%)
 Primary tibial screw type 77
  Bioabsorbable 41 (53.2%)
  Metal 26 (33.8%)
  PEEK 10 (13.0%)
A
CL = anterior cruciate ligament, ACLR = anterior cruciate ligament reconstruction, ALLO = allograft, BTB = bone-patellar tendon-bone, CI 2= confidence interval, HG = hamstring graft, ITB = iliotibial band, PEEK = polyetheretherketone, PT = patellar tendon, TA = tibialis anterior

Agreement

OME and chart-reviewed data comparison showed “near perfect” (kappa ≥ 0.81) or “substantial” to “almost perfect” agreement (CCC > 0.95) for all data tested (Table 2). The highest agreement level among nominal variables occurred in the reporting of graft type (kappa = 1.000, 95% CI [1.000, 1.000]), medial meniscus implant system (kappa = 1.000, 95% CI [1.000, 1.000]), lateral meniscus implant system (kappa = 1.000, 95% CI [1.000, 1.000]), and lateral meniscus treatment (kappa = 1.000, 95% CI [1.000, 1.000]). The lowest degree of agreement occurred in reporting of lateral meniscus status (kappa = 0.859, 95% CI [0.759, 0.960]). For the three numeric variables, the CCC was lowest for femur tunnel size (CCC = 0.977, 95% CI [0.964, 0.985]) and highest for graft strand number (CCC = 0.998, 95% CI [0.997, 0.999]).

Table 2 - Agreement Between Chart-Reviewed and OrthoMiDaS Episode of Care Data
Measure n Agreement Statistic 95% Confidence Interval
Operative limb 100 0.940 0.873, 1.000
ACL
 Status 100 0.904 0.716, 1.000
 Graft type 93 1.000 1.000, 1.000
 Graft strand number 75 0.998a 0.997, 0.999
 Femur tunnel size 79 0.977a 0.964, 0.985
 Primary femoral fixation type 94 0.961 0.908, 1.000
 Femoral screw type 31 0.871 0.699, 1.000
 Tibial tunnel size 79 0.990a 0.984, 0.993
 Primary tibial fixation type 93 0.925 0.821, 1.000
 Tibial screw type 76 0.934 0.860, 1.000
Medial meniscus
 Status 100 0.898 0.812, 0.985
 Treatment 37 0.952 0.860, 1.000
 Implant system 18 1.000 1.000, 1.000
Lateral meniscus
 Status 100 0.859 0.759, 0.960
 Treatment 46 1.000 1.000, 1.000
 Implant system 5 1.000 1.000, 1.000
A
CL = anterior cruciate ligament
a
Agreement between numeric variables is measured through the concordance correlation coefficient.

Provider Completion Rate

Graft strand number was reported in OME in 100% of the ACLR patients, but only in 80% of the ACLR patients in the operative reports. Femur tunnel size was reported in OME in 100% of the ACLR patients, but only in 85% of the ACLR patients in the operative reports. Tibial tunnel size was reported in OME in 100% of the ACLR patients, but only in 85% of the ACLR patients in the operative reports (Table 3).

Table 3 - Provider Data Completion Rates
Measure Operative Report OrthoMiDaS Episode of Care Database P Value
n % Completion n % Completion
ACL
 Graft strand number 75 80 94 100 P < 0.001
 Femur tunnel size 80 85 94 100 P < 0.001
 Tibial tunnel size 80 85 94 100 P < 0.001
A
CL = anterior cruciate ligament

Time

The median provider time to complete the data entry for a single patient was approximately 2 minutes.

Discussion

Agreement

Although all data tested showed near-perfect agreement, ACL status (kappa = 0.904), lateral meniscus status (kappa = 0.898), femoral screw type (kappa = 0.871), and medial meniscus status (kappa = 0.859) displayed the least agreement of all data tested. Despite the obvious need for accuracy in determining these parameters, a high degree of variability exists in surgeons' operative dictation description of these lesions and implants. Some surgeons provided detailed, quantitative descriptions (eg, 5-mm, full-thickness, radial tear in the posterior horn), whereas other surgeons gave very brief descriptions (eg, small tear in the lateral meniscus), leading to the inability to extract accurate data from the operative report.

Improved Completion Rate

Historically, the operative report is used as a key data source for most orthopaedic retrospective research. Despite widespread use, the quality of these data is limited.26 Operative reports tend to infrequently report quantitative data, markedly limiting the precision of research and quality measurement.13 Moreover, late dictation of reports, or dictation by residents, may also increase error rates.27,28 This error is compounded by errors in data extraction from the medical record itself. This limited quality of data contributes to the large gap, both in validity and reliability, between prospective and retrospective research.29 Together, these limitations of operative report data delay the progress of the field and cause increased cost burden.

Some improvements in the quality of capturing surgical data have been previously made. Customized, computerized, templated operative reporting systems are available and can improve the consistency of reporting of key operative parameters.30 The branching logic of OME not only improves the completeness of reporting operative parameter details but also maximizes efficiency in data input. Such systems can dramatically decrease completion delays and reduce the cost of surgical documentation.31

Before this study, templates and computer assistance are effective in increasing the completeness of operative report data.30 This finding was consistent with our findings that graft strand number, femur tunnel size, and tibial tunnel size were all more frequently collected in the OME data capture system. The OME user interface incorporates dropdown menus that likely function as memory aids for key surgical parameters.30 The OME computer defaults and branching logic will not accept missing key risk factors identified in the most recent ACLR and meniscal outcomes research.6,18-20 Branching logic and the smartphone interface speed the data entry process. Finally, an automated e-mail reminder system is used to ensure the high data capture rate.

Limitations

Currently, although the operative report is widely used, no true benchmark exists for operative reporting. Thus, the absolute accuracy of the OME database (ie, OME data compared directly with true occurrences in the operating room) is difficult to assess. However, the present study indicates OME data are consistent with the current most pervasive methodology and demonstrate less information loss.

Conclusion

The OME data capture system demonstrated “almost perfect” agreement (kappa ≥ 0.81) on all 13 nominal variables and “substantial” agreement (CCC > 0.95) to “almost perfect” agreement (CCC > 0.099) on all three numeric variables tested. In addition, the OME data capture system improved the reporting of key operative parameters necessary for outcomes research and internal quality improvement (100% for all data tested). Furthermore, the developers of this system continue to develop branching logic and data capture tools for additional high-volume orthopaedic surgical procedures. Moreover, widespread use of this system can bring the highest level of prospective quality data to everyday surgical practice. Finally, this technology could potentially replace standard narrative/dictation-based operative reporting, and transform observational and retrospective orthopaedic clinical research, to a prospective cohort model.

Appendix 1 OrthoMiDaS Episode of Care Basic Meniscal Data Fields

-
Data Field Dropdown Selections
Meniscus status Normal
Complete tear
Partial tear
Previous partial excision
Previous repair
New tear None
Partial
Complete
Main tear type Oblique/flap
Longitudinal
Bucket-handle
Radial
Root
Horizontal
Complex
Location to horns Anterior only
Posterior only
Both
Location to blood supply Red-red
Red-white
White-white
Main tear extent Partial, periphery intact
Complete to periphery
Main tear length (mm) <6 mm
6–10 mm
11–15 mm
16–20 mm
>20 mm
Treatment No treatment for tear
Partial excision
Repair
Abrade + trephine
Meniscal transplant
Repair technique All inside
Inside-out
Both inside-out and all in
Outside-in
Number of inside-out and/or outside-in sutures Arthrex
Biomet
Covidien
CONMED
Cayenne
DePuy Mitek
Smith & Nephew
Others
Implant system Select from preprogrammed implant system library specific to manufacturer selection
Number of implants Select integer value between 0 and 10
Discoid meniscus No
Yes, partial
Yes, complete

References

References printed in bold type are those published within the past 5 years.

1. Porter ME, Larsson S, Lee TH: Standardizing patient outcomes measurement. N Engl J Med 2016;374:504-506.
2. Ahlden M, Samuelsson K, Sernert N, Forssblad M, Karlsson J, Kartus J: The Swedish National Anterior Cruciate Ligament Register: A report on baseline variables and outcomes of surgery for almost 18,000 patients. Am J Sports Med 2012;40:2230-2235.
3. Lind M, Menhert F, Pedersen AB: Incidence and outcome after revision anterior cruciate ligament reconstruction: Results from the Danish registry for knee ligament reconstructions. Am J Sports Med 2012;40:1551-1557.
4. Paxton EW, Namba RS, Maletis GB, et al.: A prospective study of 80,000 total joint and 5000 anterior cruciate ligament reconstruction procedures in a community-based registry in the United States. J Bone Joint Surg Am 2010;92(Suppl 2):117-132.
5. Lynch TS, Parker RD, Patel RM, et al.: The impact of the Multicenter Orthopaedic Outcomes Network (MOON) research on anterior cruciate ligament reconstruction and orthopaedic practice. J Am Acad Orthop Surg 2015;23:154-163.
6. Wright RW, Huston LJ, Haas AK, et al.: Effect of graft choice on the outcome of revision anterior cruciate ligament reconstruction in the Multicenter ACL Revision Study (MARS) cohort. Am J Sports Med 2014;42:2301-2310.
7. Pugely AJ, Martin CT, Harwood J, Ong KL, Bozic KJ, Callaghan JJ: Database and registry research in orthopaedic surgery: Part I: Claims-based data. J Bone Joint Surg Am 2015;97:1278-1287.
8. Pugely AJ, Martin CT, Harwood J, et al.: Database and registry research in orthopaedic surgery: Part 2: Clinical registry data. J Bone Joint Surg Am 2015;97:1799-1808.
9. Mather RC, Hettrich CM, Dunn WR, et al.: Cost-effectiveness analysis of early reconstruction versus rehabilitation and delayed reconstruction for anterior cruciate ligament tears. Am J Sports Med 2014;42:1583-1591.
10. Boyer P, Villain B, Pelissier A, et al.: Current state of anterior cruciate ligament registers. Orthop Traumatol Surg Res 2014;100:879-883.
11. Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P: Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: A multi-site assessment. EGEMS (Wash DC) 2014;2:1079.
12. Pan L, Pan L, Fergusson D, Schweitzer I, Hebert PC: Ensuring high accuracy of data abstracted from patient charts: The use of a standardized medical record as a training tool. J Clin Epidemiol 2005;58:918-923. http://journals.ohiolink.edu/ejc/article.cgi?issn=08954356&issue=v58i0009&article=918_ehaodamraatt. Accessed July 15, 2016.
13. Scherer R, Zhu Q, Langenberg P, Feldon S, Kelman S, Dickersin K: Comparison of information obtained by operative note abstraction with that recorded on a standardized data collection form. Surgery 2003;133:324-330.
14. Spindler KP, Huston LJ, Wright RW, et al.: The prognosis and predictors of sports function and activity at minimum 6 years after anterior cruciate ligament reconstruction: A population cohort study. Am J Sports Med 2011;39:348-359.
15. Fabricant PD, Robles A, Downey-Zayas T, et al.: Development and validation of a pediatric sports activity rating scale: The Hospital for Special Surgery Pediatric Functional Activity Brief Scale (HSS Pedi-FABS). Am J Sports Med 2013;41:2421-2429.
16. Selim AJ, Rogers W, Fleishman JA, et al.: Updated U.S. population standard for the Veterans RAND 12-Item Health Survey (VR-12). Qual Life Res 2009;18:43-52.
17. Roos EM, Lohmander LS: The Knee injury and Osteoarthritis Outcome Score (KOOS): From joint injury to osteoarthritis. Health Qual Life Outcomes 2003;1:64.
18. Kaeding CC, Pedroza AD, Reinke EK, Huston LJ, Spindler KP: Risk factors and predictors of subsequent ACL injury in either knee after ACL reconstruction: Prospective analysis of 2488 primary ACL reconstructions from the MOON cohort. Am J Sports Med 2015;43:1583-1590.
19. Katz JN, Brophy RH, Chaisson CE, et al.: Surgery versus physical therapy for a meniscal tear and osteoarthritis. N Engl J Med 2013;368:1675-1684.
20. Cox CL, Huston LJ, Dunn WR, et al.: Are articular cartilage lesions and meniscus tears predictive of IKDC, KOOS, and Marx activity level outcomes after anterior cruciate ligament reconstruction?: A 6-year multicenter cohort study. Am J Sports Med 2014;42:1058-1067.
21. Shah J, Rajgor D, Pradhan S, McCready M, Zaveri A, Pietrobon R: Electronic data capture for registries and clinical trials in orthopaedic surgery: Open source versus commercial systems. Clin Orthop Relat Res 2010;468:2664-2671.
22. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG: Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377-381.
23. Viera AJ, Garrett JM: Understanding interobserver agreement: The kappa statistic. Fam Med 2005;37:360-363.
24. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174.
25. McBride GB: A proposal of strength of agreement criteria for Lin's concordance correlation coefficient: NINA Client Report 2005, New Zeal, Natl Inst Water Atmos Res Ltd Hamilton, 2005, pp 1-10.
26. Severn Audit and Research Collaborative in Orthopaedics (SARCO), Blackburn J: Assessing the quality of operation notes: A review of 1092 operation notes in 9 UK hospitals. Patient Saf Surg 2016;10:5.
27. Novitsky YW, Sing RF, Kercher KW, Griffo ML, Matthews BD, Heniford BT: Prospective, blinded evaluation of accuracy of operative reports dictated by surgical residents. Am Surg 2005;71:627-631.
28. ten Broek RPG, van den Beukel BAW, van Goor H: Comparison of operative notes with real-time observation of adhesiolysis-related complications during surgery. Br J Surg 2013;100:426-432.
29. Nagurney JT, Brown DFM, Sane S, Weiner JB, Wang AC, Chang Y: The accuracy and completeness of data collected by prospective and retrospective methods. Acad Emerg Med 2005;12:884-895.
30. Gur I, Gur D, Recabaren JA, et al.: The computerized synoptic operative report. Arch Surg 2012;147:71.
31. Laflamme MR, Dexter PR, Graham MF, Hui SL, McDonald CJ: Efficiency, comprehensiveness and cost-effectiveness when comparing dictation and electronic templates for operative reports. AMIA Annu Symp Proc 2005;2005:425-429.
Copyright © 2019 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the American Academy of Orthopaedic Surgeons.