Secondary Logo

Journal Logo

Readability of Sports Injury and Prevention Patient Education Materials From the American Academy of Orthopaedic Surgeons Website

Minoughan, Chelsea, BS; Schumaier, Adam, MD; Kakazu, Rafael, MD; Grawe, Brian, MD

JAAOS Global Research & Reviews: March 2018 - Volume 2 - Issue 3 - p e002
doi: 10.5435/JAAOSGlobal-D-18-00002
Research Article
Open

Introduction: The purpose of this study is to evaluate the readability of 114 Sports Injury and Prevention patient education materials provided by the American Academy of Orthopaedic Surgeons (AAOS).

Methods: We evaluated all articles written in English posted under the Sports Injury and Prevention section of the AAOS website using readability software to compute six readability scores, which we compared with the eighth-grade level using a two-tailed one-sample Student t-test.

Results: The mean reading grade level calculated by each readability test was markedly higher than the eighth-grade level. We reported mean ± SD for each test: Flesch-Kincaid grade level (8.95 ± 1.51; P < 0.001), Simple Measure of Gobbledygook (11.53 ± 1.18; P < 0.001), Coleman-Liau index (11.16 ± 1.33; P < 0.001), Gunning Fog index (11.06 ± 1.63; P < 0.001), New Dale-Chall (9.49 ± 1.66; P < 0.001), and FORCAST formulas (10.96 ± 0.60; P < 0.001).

Discussion: This study shows that patient education materials provided by the AAOS concerning sports injury and prevention are written at a readability level too high for patients to understand. On average, patient materials are written at least 2.5 grade levels higher than national recommendations. Only 7% of the 114 articles had readability scores in line with national recommendations. These findings indicate a need for revised patient education materials geared toward bringing the readability level down to the recommended eighth-grade level.

From the Department of Orthopaedics and Sports Medicine, University of Cincinnati, Cincinnati, OH.

None of the following authors or any immediate family member has received anything of value from or has stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this article: Ms. Minoughan, Dr. Schumaier, Dr. Kakazu, and Dr. Grawe.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

The internet is a tremendous tool for providing patients with quick access to health information and has reportedly been used by >50% of Americans with internet access for such a purpose.1,2 As a result, many orthopaedic specialty societies have provided patient education materials pertaining to common injuries and diagnoses on their websites.1,3-8 This allows patients to learn more about their diagnoses and the treatment options available to them from a reliable source. Although this information is accurate and readily available to patients, the content provided on these websites may not be understood by patients if the material is written at a reading level higher than that which the patient can understand.

This reading level discrepancy in patient education materials exists across all fields of medicine, despite the recommendation from multiple organizations, such as the National Institutes of Health and the National Academy of Medicine, to provide education materials to the lay public at the eighth-grade reading level.9,10 Analysis of materials provided to patients by the American College of Cardiology and the American Heart Association showed that articles were written at a reading level markedly higher than the National Institutes of Health recommendation.11 A similar problem has been identified regarding oncology and palliative care materials.12,13 Similar trends are seen among the orthopaedic literature.1,3-8,14

The goal of this study is to evaluate the sports medicine patient education materials provided by the American Academy of Orthopaedic Surgeons (AAOS). In 2008 and 2014, the patient education materials provided on the AAOS website were assessed for readability and showed that most of the articles were written above the nationally recommended level.3,6 It has been three years since the previous assessment, and 25% of the sports injury articles on the AAOS website have been edited in that time. This study will determine the current reading grade level of the patient materials provided on the AAOS website, allowing us to determine the progress made since the previous assessments. We hypothesize that, on average, the readability score of the AAOS English-language articles will exceed the nationally recommended eighth-grade reading level.

Back to Top | Article Outline

Methods

The reading level required for a patient to understand written material is termed “readability” and is typically yielded in a grade-level format to specify the level of education needed to understand the document.3,15 All English-language patient-directed articles from the AAOS website pertaining to sports injuries and prevention as of June 2017 were included in this study. This included articles within the Sports Injuries and Prevention subsection of the Health Centers portion of the OrthoInfo website (http://www.orthoinfo.org/) from the AAOS. Articles that contained only hyperlinked information were excluded from this study. One hundred twenty-six articles were initially identified from the Sports Injuries & Prevention subsection of the AAOS website. Eleven articles were not written in English and were excluded from this study. One article contained only hyperlinks to more information, and it was also excluded from the study, leaving 114 articles to be analyzed.

Each article's content was pasted as plain text into a new Microsoft Word (Microsoft) document. All figures, figure legends, copyright notices, disclaimers, acknowledgements, citations, references, and hyperlinks were excluded from the analyzed material in accordance with software guidelines.11 The Microsoft Word documents were then analyzed using the Readability Studio Professional Edition, version 2015 software package (Oleander Software). Because there is currently no benchmark test for readability, it is recommended to use more than one readability formula to increase the validity of the tests.15 The software calculated six different formulas to determine six readability scores for each article. The six readability formulas used were the Flesch-Kincaid grade level, the Simple Measure of Gobbledygook index, the Coleman-Liau index, the Gunning Fog index, the New Dale-Chall formula, and the FORCAST readability formula. These six readability tests were chosen because they have been previously used for the analysis of patient health education materials.4,6,11,12,15Table 1 shows the formula used by the software to make each calculation.12

Table 1

Table 1

The mean reading level for each article was determined by taking the average of the six calculated readability scores, with each readability assessment receiving equal weight in the analysis. This is similar to the analysis performed in previous readability studies.6 The overall average and SD was also calculated for each of the six readability statistics. The average calculated by each readability test statistic was then compared with the nationally recommended eighth-grade reading level using a two-tailed, one-sample Student t-test, assuming unequal variances. The software also calculated the number of long words and complex words in each article. Any word made up of six or more characters was considered a long word, whereas any word with three or more syllables was considered a complex word.

Back to Top | Article Outline

Results

Readability Studio software successfully calculated the Flesch-Kincaid grade level, the Simple Measure of Gobbledygook index, the Coleman-Liau index, the Gunning Fog index, the New Dale-Chall formula, and the FORCAST readability formula for all 114 articles input for analysis. Of the 114 articles analyzed, none were written below the seventh-to-eighth-grade reading level, and only four articles were written at the seventh-to-eighth-grade reading level. Figure 1 shows the number of articles written at each grade level based on the average of the eight calculated readability scores.

Figure 1

Figure 1

Table 2 shows the average reading grade level and SD calculated by each of the six readability tests for the sports injury and prevention materials provided on the AAOS website. When we compared each readability test average against the nationally recommended eighth-grade reading level, we found that each of the six tests showed that the material was written at a markedly higher reading level than the recommendation.

Table 2

Table 2

The readability software also shows the percentage of complex words (any word containing three or more syllables) and long words (any word containing six or more characters) for each article analyzed as the output. Higher percentages of complex and long words increase the reading grade level of the article. The average percentage of complex words in the analyzed articles was 16.3%, with an SD of 3.3%, whereas the average percentage of long words in the analyzed articles was 37.2%, with an SD of 3.8%. Table 3 shows problematic words identified by the readability software and suggestions for replacement terms that would be more understandable to decrease the reading level of the document.

Table 3

Table 3

Back to Top | Article Outline

Discussion

With many patients today searching the internet for information related to their health, it is important to ensure that information is not only accurate and accessible but also written at a reading level that the adult population can understand.1,2 Specifically, in the field of orthopaedics, many authors have reported patients using the internet to learn more about their diagnosis, some as high as 75%.3,16,17 Previous analysis of information provided by the AAOS through the OrthoInfo website in 2008 and 2014 showed that most of the materials were written at a reading level too complex for patients to understand.3,6 This led to our hypothesis that the patient education material about sports injury and prevention provided on the OrthoInfo website would also be written above the recommended eighth-grade reading level. Our analysis using six different readability tests showed that the articles were written at a level markedly higher than the 8th-grade level, with eight of the articles written above the 12th-grade reading level.

In 2008, the mean grade level of the patient education materials on the AAOS website was reported as a 10.4-grade level using the Flesch-Kincaid readability assessment. At that time, 85% of the articles were written above the eighth-grade reading level.3 Another analysis of the AAOS patient education materials, conducted in 2014, reported that the mean Flesch-Kincaid grade level had dropped to 9.3, but 84% of the articles were still written above the eighth-grade reading level.6 The following results from this study were calculated using only the Flesch-Kincaid grade level to accurately compare our findings with both the 2008 and 2014 findings, which were only reported as the Flesch-Kincaid grade level. Our study showed an average Flesh-Kincaid grade level of 8.95 ± 1.51, with 72% of the articles written above the eighth-grade reading level. Although these studies show that the reading level of patient education materials is trending toward the eighth-grade recommendation, most of the articles provided to patients are still written above this recommendation. Our findings are consistent with those from many studies showing that the online orthopaedic patient education material is written at grade levels too high for patients to understand.1,36,8,14 These studies show that despite progress in providing readily accessible and accurate information to patients, the same progress has not been seen in making the writing understandable for the average patient reading level.

Previous assessments of literacy in the US population conducted by the National Center for Education Statistics reported that 43% of the population has basic (29%) or below basic (14%) literary skills. Basic literacy skills allow the patient to function in typical everyday activities but do not include complex literacy activities. Only 13% of the population had literacy skills proficient for handling complex literacy activities.18,19 Patients with a lower literacy level have reported feeling shameful of this fact and are less likely to ask clarifying questions or admit to physicians that they do not understand something in the office.20-22 Therefore, many patients seek online information to answer questions they have about their diagnosis. Many reputable organizations, such as the AAOS, provide reliable education material for patients. Problems can arise if patients cannot understand the material. The information provided could be misinterpreted, or the patient may search for alternate sources that are more easily understood but less reliable. Both scenarios result in a misinformed patient, which may threaten a patient's ability to make informed decisions about their care.7,23

The discrepancy shown in this study between the reading level of patient education materials and the national recommendation is seen across many fields of medicine and indicates that changes need to be made to improve the patient's ability to understand the provided health information.1,3-8,11-14 To improve the material analyzed in our study, we looked at the variables used to determine the readability of a document. Two variables that increase the reading grade level of a document are the percentage of long and complex words in the document. The articles in this study were found to have an average of 16.3% of complex words and 37.2% of long words in each article. Decreasing the percentage of these words would lower the reading level. Some of the problematic terms and acceptable substitutions that would decrease the reading level of the document are indicated in Table 3.

This study is not without limitations. All images were removed from the material analyzed by the readability software because the readability tests used in this study are not capable of evaluating images. Figures can enhance the comprehension of a document; however, the only readability tool designed to analyze figures in a document is the Suitability Assessment of Materials, which has been criticized for its subjectivity and lack of validation.5,7 We excluded articles not written in English because, like figures, the readability tests used in this study are not capable of evaluating multiple languages. This excluded nine articles written in Spanish and two articles written in Portuguese; however, this likely did not affect the overall analysis. With 21.4% of the US population speaking languages other than English, additional research must be conducted to evaluate the patient materials provided in other languages.24

Although it is important that the educational material be readable, there are several additional prerequisites for effective communication, including validity and accessibility, that were not evaluated in this study. A study examining the effectiveness of providing preoperative information sheets for patients who undergo spinal surgery demonstrated that patients have a better overall comprehension of medical information when information is presented in writing compared with information provided orally by the surgeon.25 Our study showed that the reading level of the information provided on the AAOS website is currently too high. But, additional studies aimed at determining the validity of the information and measuring the patient's comprehension of the material are required to accurately determine how well patients understand the material presented in this format. It should be noted that accessing online materials may be a challenge for some patients if they do not have internet access to the internet or if they have impaired vision or other limitations that affect their ability to read the patient education material provided on the AAOS website. For this reason, surgeons must be cognizant of their patients' limitations and the resources they have available. For example, if patients do not have internet access, the surgeon could print the material for them.

Our study demonstrates that patient education materials provided by the AAOS concerning sports injury and prevention are written at a readability level too high for patients to understand. These findings indicate a need for revision of patient education materials to achieve the eighth-grade reading level. In the future, before new materials are published, readability analysis should be used to ensure that the content is written appropriately for the recommended patient reading level. Our recommendations for improving readability are to decrease the amount of complex and long words in the text. Previous readability studies have provided additional ideas for improving patient materials.3,4,26 Although our recommendation is to decrease the reading level of the patient education materials, we acknowledge that it is important to accurately convey information without oversimplifying education materials. The eighth-grade reading level is thought to align with most patients' capabilities without over simplifying the material. Information provided for patients must match the comprehension capabilities of the average patient and be designed to increase the knowledge of a diagnosis and understand the various treatment options.

Back to Top | Article Outline

References

1. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA: Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg 2010;92:1612–1618.
2. Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW: Patients' use of the internet for medical information. J Gen Intern Med 2000;17:180–185.
3. Sabharwal S, Babarudeen S, Kunju SU: Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res 2008;466:1245–1250.
4. Wang SW, Capo JT, Orillaza N: Readability and comprehensibility of patient education material in hand-related web sites. J Hand Surg Am 2009;34:1308–1315.
5. Polishchuk DL, Hashem J, Sabharwal S: Readability of online patient education materials on adult reconstruction web sites. J Arthroplasty 2012;27:716–719.
6. Roberts H, Zhang D, Dyer GS: The readability of AAOS patient education materials: Evaluating the progress since 2008. J Bone Joint Surg Am 2016;98:e70.
7. Beutel BG, Danna NR, Melamed E, Capo JT: Comparative readability of shoulder and elbow patient education materials within orthopaedic websites. Bull Hosp Jt Dis 2015;73:249–256.
8. Sambandam SN, Priyanka P, Ilango B, Ramasamy V: Quality analysis of patient information about knee arthroscopy on the world wide web. Arthroscopy 2007;23:509–513.
9. National Institutes of Health: How to Write Easy-to-Read Health Materials: MedlinePlus. 2017. https://medlineplus.gov/etr.html. Accessed July 27, 2017.
10. Nielsen-Bohlman L, Panzer AM, Kindig DA, Institute of medicine (U.S.). Committee on Health Literacy: Health Literacy: A Prescription to End Confusion. Washington, DC: National Academies Press, 2004.
11. Kapoor K, George P, Evans MC, Miller WJ, Liu SS: Health literacy: Readability of ACC/AHA online patient education material. Cardiology 2017;138:36–40.
12. Prabhu AV, Crihalmeanu T, Hansberry DR, et al: Online palliative care and oncology patient education resources through Google : Do they meet national health literacy recommendations? Pract Radiat Oncol 2017;7:306–310.
13. Tran BNN, Singh M, Singhal D, Rudd R, Lee BT: Readability, complexity, and suitability of online resources for mastectomy and lumpectomy. J Surg Res 2017;212:214–221.
14. Jawa A: Readability of arthroscopy-related patient education surgeons and arthroscopy association of North America. Arthrosc J Arthrosc Relat Surg 2013;29:1108–1112.
15. Babarudeen S, Sabharwal S: Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res 2010;468:2572–2580.
16. Brooks BA: Using the Internet for patient education. Orthop Nurs 2001;20:69–77.
17. Beall MS, Beall MS, Greenfield ML, Biermann JS: Patient Internet use in a community outpatient orthopaedic practice. Iowa Orthop J 2002;22:103–107.
18. Kirsch IS, Jungeblut A, Jenkins L, Kolstad A, Secretary RW: National Center for Education Statistics: Adult literacy in America. 2002;202–219.https://nces.ed.gov/pubs93/93275.pdf. Accessed July 9, 2017.
19. National Center for Education Statistics: National Assessment of Adult Literacy (NAAL). https://nces.ed.gov/naal/index.asp. Accessed July 19, 2017.
20. Parikh NS, Parker RM, Nurss JR, Baker DW, Williams MV: Shame and health literacy: The unspoken connection. Patient Educ Couns 1996;27:33–39.
21. Menendez ME, van Hoorn BT, Mackert M, Donovan EE, Chen NC, Ring D: Patients with limited health literacy ask fewer questions during office visits with hand surgeons. Clin Orthop Relat Res 2017;475:1291–1297.
22. Katz MG, Jacobson TA, Veledar E, Kripalani S: Patient literacy and question-asking behavior during the medical encounter: A mixed-methods analysis. J Gen Intern Med 2007;22:782–786.
23. Rubin MA: The collaborative autonomy model of medical decision-making. Neurocrit Care 2014;20:311–318.
24. United States Census Bureau: American FactFinder: Selected social characteristics in the United States. 2012-2016 American Community Survey 5-year estimates. https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_16_5YR_DP02&src=pt. Accessed July 9, 2017.
25. Madkouri R, Grelat M, Vidon-Buthion A, Lleu M, Beaurain J, Mourier K-L: Assessment of the effectiveness of SFCR patient information sheets before scheduled spinal surgery. Orthop Traumatol Surg Res 2016;102:479–483.
26. Badarudeen S, Sabharwal S: Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America Web Sites. J Bone Joint Surg Am 2008;90:199–204.
Copyright © 2018 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the American Academy of Orthopaedic Surgeons