Secondary Logo

Journal Logo

Original Article

Readability of Online Patient Education Materials From the AAOS Web Site

Sabharwal, Sanjeev MD1,a; Badarudeen, Sameer MBBS1; Unes Kunju, Shebna MBBS1

Author Information
Clinical Orthopaedics and Related Research: May 2008 - Volume 466 - Issue 5 - p 1245-1250
doi: 10.1007/s11999-008-0193-8
  • Free



In 2000, the American Academy of Orthopaedic Surgeons (AAOS) introduced the Internet-based patient education database, “Your Orthopaedic Connection [3].” Every day more than 35,000 people visit this Web site to access education materials on diverse orthopaedic conditions [2] (oral communication, Jim Ogale, AAOS Web site staff, June 6, 2007). The goal of this Web site is to enhance patient-physician communication by providing validated and up-to-date information about various orthopaedic conditions in a way that is “sensitive to diversity and readability and to strengthen the bond between physicians and patients [24].”

Readability of a text is the reading comprehension level a person must have to understand the written material and is an important determinant of a person's ability to comprehend health information [1, 14, 38]. The Flesch-Kincaid (FK) grade formula is one of the most commonly used tools to assess the readability of written materials in terms of the academic grade [1, 14]. A higher FK grade level of a text indicates a greater level of difficulty to read and comprehend the material and thus requires more advanced reading skills than would be required of a text with a lower FK grade level. Organizations like the National Institutes of Health, the National Work Group on Cancer and Health, and the American Medical Association have recommended the readability of patient education materials should be no higher than the sixth-grade level [15, 34, 35]. The average readability of the US adult population is at the eight-grade level [18]. However, several studies suggest the readability of patient education materials in most medical specialties is beyond these recommended levels [1, 14, 18, 34]. We recently assessed the readability scores of patient education materials that were available at the AAOS and Pediatric Orthopaedic Society of North America (POSNA) Web sites, pertaining exclusively to pediatric orthopaedic conditions [5]. Only 2% of the articles available in 2007 had a FK readability score of the sixth grade level or less. It is unknown whether there is a trend toward greater readability on these two sites, but if the readability of articles remains high over a period of time, it would suggest lack of awareness of the importance of readability in the orthopaedic community.

Based on the cited reports of relatively high reading levels of patient education material, we hypothesized the readability level of the majority of patient education materials available on the AAOS Web site would be higher than the recommended FK grade level of sixth grade. We then asked whether there was a trend toward lower readability scores of articles available on the Web site with time. Finally, we asked whether the variability of the readability level of articles would differ between subject categories in the patient education Web site.

Materials and Methods

We searched the patient education database, “Your Orthopaedic Connection,” of the AAOS Web site [3] during April 2007 and downloaded all 663 patient education articles. Materials had been created between September 1999 and July 2006. We excluded articles written in a language other than English. We noted the subject category of each article and the date of the last update and saved the text as a separate Microsoft® Word® (Microsoft Corp, Redmond, WA) document. The text was copied as plain text format to avoid HTML tags [30]. Any information related to Web page navigation, copyright notices, postal addresses, phone numbers, uniform resource locaters (URLs), disclaimers, date stamps, author information, citations, references, feedback questionnaires, or hyperlinks were omitted. Followup editing was performed as recommended by Flesch and others [19, 21] by removing decimal points from numbers and colons and semicolons in sentences. The FK grade formula is best suited for text arranged in a paragraph as opposed to list format [32]. A running passage containing a minimum of 35 words was defined as a paragraph. To get the most representative sample, similar to methods adopted by other researchers, the three longest paragraphs of each patient education article were selected [1, 17]. Our Institutional Review Board granted a waiver for the study.

There were 663 articles grouped under 21 subject categories in the AAOS patient information Web site, “Your Orthopaedic Connection [3].” Forty of these articles were in Spanish and thus excluded from the study. Of the remaining 623 articles, 573 articles in 19 subject categories met the study's inclusion criteria. Of the 573 articles, 147 were listed in more than one subject category, resulting in 426 unique articles.

The FK grade level of readability was assessed using Microsoft® Office Word® software (2003 Service Pack 2; Microsoft Corp). The built-in tool to measure readability is disabled by default, and the user has to enable it by sequentially selecting the commands “Tools,” “Options,” “Spelling and Grammar,” and then enabling the option “Show readability statistics” followed by clicking on the icon for “Spelling and Grammar” from the tool bar. The underlying formula for determining the FK grade level is as follows [21, 27]: (0.39 × average number of words per sentence) + (11.8 × average number of syllables per word) - 15.59.

The same individual (SUK) calculated all FK grade levels. Interobserver reliability was assessed by calculating the intraclass correlation coefficient using 30 randomly selected articles that were graded by another individual (SB). An intraclass correlation coefficient of 0 to 0.24 reflects poor correlation; 0.25 to 0.49, low; 0.50 to 0.69, fair; 0.7 to 0.89, good; and 0.9 to 1.0, excellent [33]. The intraclass correlation coefficient for assessing FK grade level was 0.96, indicating excellent interobserver reliability.

The mean and 95% confidence interval values of the FK grade level were calculated. Using descriptive statistics and analysis of variance, the readability grade scores of articles grouped under different subject categories were analyzed. A two-sample t test was used to compare the FK grade for articles in each subject category against the rest of the articles. The Pearson correlation coefficient (r) was calculated to study the relationship of the FK grade of articles with the date when the latest version of the article was available online. Statistical analysis was performed using the SAS® software (Version 9.1; SAS Institute Inc, Cary, NC).


The majority (98%) of the 426 unique articles on the patient education Web site had readability scores that were higher than the sixth-grade level. Only 10 articles (2%) had the recommended readability level of the sixth-grade level or less [15, 34, 35]. The mean FK grade level of the 426 articles was 10.4 (95% confidence interval, 10.2-10.6). Moreover, 85% of the articles had readability above the eighth-grade level (Fig. 1).

Fig. 1:
The distribution of FK readability grades of 426 unique articles that were available at the AAOS patient education Web site, “Your Orthopaedic Connection,” [3] is illustrated.

The readability level did not change with time (r = 0.0003) (Fig. 2). Thus the readability of articles remained high throughout the entire period.

Fig. 2:
A scatterplot displays the lack of relationship of the FK grade of the online articles with the date when the latest version of the article was available online. Pearson correlation coefficient r = 0.0003 (n = 426).

Although none of the 19 subcategories of articles achieved a mean FK grade level of sixth or lower (Table 1), we found substantial variation in the readability scores among the patient education articles in different subcategories. The FK grades of articles in the Hand and Foot sections were lower (p = 0.001 and p = 0.028, respectively), indicating easier readability, whereas the FK grades of articles in the General Information and Patient Stories categories were higher (p = 0.006 and p < 0.001, respectively) than the remaining articles.

Table 1:
Flesch-Kincaid grade level of AAOS patient education articles (n = 573) in 19 subject categories


There is increasing concern regarding the disparity between the readability levels of patient education materials available online and the reading skills of the target population [1, 14, 18]. Based on our review of the literature and our recent report dealing exclusively with the readability scores of online pediatric orthopaedic materials [5], the readability of medical information is higher than generally recommended. Based on this literature we hypothesized the readability level of the majority of patient education materials available on the AAOS Web site, “Your Orthopaedic Connection [3]” would be higher than the recommended FK grade level of sixth grade. We then asked whether there was a trend toward lower readability scores of articles available on the Web site with time. Finally, we asked whether the variability of the readability level of articles would differ between subject categories in the patient education Web site.

We note several limitations in our study. First, there is an inherent weakness in the assessment of readability of health-related text using the FK grade level because this tool relies solely on the number of syllables in a word and the number of words in a sentence [19]; the number of syllables may not accurately reflect reading level. In the field of medicine, the unfamiliarity of nonprofessionals to medical terms, even if they are short, such as “lupus,” “physis,” and “colon,” can lead to underestimation of the reading skills required to fully comprehend medical literature with the use of the FK grading system [23]. Although comprehension of a given material may be enhanced by the addition of illustrative figures, improved layout, and appropriate use of font size and color [34], the FK grading tool does not assess these features. Other instruments such as the Suitability Assessment of Materials (SAM), a relatively new tool, can assess these factors to measure the comprehensibility of patient education materials [7, 18, 25, 37]. However, SAM is not as validated as the FK grade score, is more time-consuming [18], and is “inherently subjective [37].” Furthermore, the scoring system by SAM is not based on grades and thus is incompatible with the recommendations by healthcare organizations [18]. Second, we did not directly assess the reading skills of our patient population. There are many validated tools available to assess the reading skills of a given population. These include the Wide Range Achievement Test, Rapid Estimate of Adult Literacy in Medicine, and the Test of Functional Health Literacy in Adults [10]. However, in the context of the current study, because the patient education materials available through the AAOS Web site are available in the public domain and do not have a definite target population, trying to match the reading level of these materials to the reading skills of our patient population may be irrelevant. Third, we excluded articles that were available in a language other than English. Given that 41% of the people who are visiting the AAOS Web site are from countries where English is not the primary language, the need to make patient education articles easier to read is even more imperative [2]. Fourth, we did not assess the entire text of the articles on the Web site, but only the longest three paragraphs. We presume these would reflect the entire article. Finally, we limited our assessment of readability of orthopaedic patient education materials to one Web site. Patients and their caretakers may access more than one Web site to gain additional insight into their orthopaedic ailments. Nevertheless, we believe our study is relevant because orthopaedic surgeons often direct their patients to the AAOS Web site to find accurate, peer-reviewed, and up-to-date information. The Web site reach as measured by number of daily unique visitors of more than 35,000 adds further credibility to our sample selection [2] (oral communication, Jim Ogale, AAOS Web site staff, June 6, 2007).

Numerous authors report patients seeking orthopaedic care extensively use the Internet as a resource for patient education [8, 9, 13, 28]. Access to the Internet and its use to obtain health information are increasing globally at a rapid pace. One study reported three of every four patients attending an orthopaedic clinic have access to the Internet [13]. The Pew Research Center report found, in 2006 alone, more than 100 million people in the United States searched the Internet to find health information about diseases from which they or their relatives and friends suffer [20]. However, patient education materials available on the Internet may not have the appropriate reading level for the average person. A survey conducted in a pediatric orthopaedic outpatient clinic reported 2/3 of the respondents believed the health information available on the Internet was “too technical [4].” However, the authors did not assess the issue of readability. Our findings highlight the need for orthopaedic surgeons and educators to recognize the concept of readability while preparing and reviewing health education materials for their patients. Although only 2% of the English articles on the AAOS Web site had readability at the sixth-grade level or lower, creating musculoskeletal education articles with easier readability for patients seems attainable.

A recent national adult literacy survey found 40 million people in the adult US population have literacy skills equivalent to less than the fifth-grade level and another 50 million have reading skills between the sixth- and eighth-grade levels [29, 36]. Reading grade level is distinct from the last academic grade achieved in schools and colleges. Patients read approximately five grades lower than their highest attained academic grade [26, 31]. The reason for this difference between academic grade and reading level may be multifactorial and possibly related to a flaw in the education system and the fact that a substantial portion of the patient population belong to the lower socioeconomic status [18]. Health literacy is the best predictor of an individual's health status [38]. It is defined as the “degree to which individuals have the capacity to obtain, process and understand health information and services needed to make appropriate health decisions [29, 36].” Low health literacy is associated with poor health status, increased hospitalization rate, poor compliance to treatment, missed appointments, and increased healthcare expenditure [6, 10, 18, 34, 36].

The improved access to the Internet is making the online population increasingly similar to the general population [20]. The fact that 33% of the “online health seekers” had only a high school diploma or less education [20] adds further proof to this assumption. Recently, a national survey found 20% of adults with “below basic health literacy'” are getting their health information from the Internet [29]. Some studies also suggest even people with good literacy skills prefer materials written in simpler format and low grade level [16, 37]. Furthermore, surgeons often are using patient education handouts printed from Web pages for their patients, and thus the readability level of Internet-based health information materials should serve the needs of all segments of society [23]. Although the data regarding readability skills of Internet users are lacking, most researchers use readability standards of the general patient population to assess the readability of online materials [11, 12, 16, 22, 23, 30, 37].

Readability of patient education materials can be assessed readily using the FK grade formula, with a relatively simple set of keystroke instructions and software available on most personal computers. The FK grade formula originally was developed for the US military in the early 1970s [27]. Since then, this instrument has been extensively validated and researched [14, 21, 31]. The advent of computers and software capable of automating the calculation has made the formula relatively simple, quick, and intuitive [14, 21, 31] and within the reach of almost every healthcare worker. As established in our study, the FK grade formula has very high interobserver reliability. Our findings support the hypothesis that the readability of patient education materials in the AAOS patient education database would be higher than the recommended level. In addition, we found the readability scores of the text did not improve with time. These findings may reflect the lack of awareness regarding the concept of readability in the orthopaedic community. Moreover, in a recent study [5], despite using a smaller and distinct set of online patient education materials dealing exclusively with pediatric orthopaedic conditions, we arrived at a similar conclusion with only 2% of the articles having readability scores of the sixth grade level or less. The method used in that study [5] also was different, in that the entire text was subject to the FK formula as opposed to assessing the readability score based on the longest three paragraphs, as in the current study.

In addition to the FK grade level available on the Microsoft® Office software, there are other software packages, such as Corel® WordPerfect® Office X3 (Corel Corp, Ottawa, Ontario, Canada), Readability Calculations (Micro Power and Light Co, Dallas, TX), Readability Studio 1.1 (Oleander Solutions, Vandalia, OH), and InText (Social Science Consulting, Rudolstadt, Germany) [35], that have readability assessment tools. However, we believe the readability score of a given text should not be the sole criterion to develop patient education materials. The “Living word vocabulary” contains approximately 43,000 words arranged in various grade levels of complexity and can be used to substitute difficult words [15, 31]. To allow improved comprehension by a larger segment of the society, simpler words can replace complex medical jargon. A list of lay terms that can be used instead of medical terms can be found at the following Web sites: and [15]. Detailed instructions on making more comprehensible patient education literature are available through the American Medical Association [38].

Ensuring patients receive education materials they can understand is the responsibility of physicians, professional organizations, and healthcare institutions [15]. Our findings suggest a substantial portion of the patient education articles available at the patient education library of the AAOS Web site have readability scores that are higher than the sixth-grade level. These findings suggest the online material presented may be too difficult for comprehension by a substantial portion of the patient population. To enhance the patient-physician dialogue, orthopaedic educators should attempt to keep the readability level of the patient education materials at a FK grade level of sixth grade or lower. Such measures will help accomplish the goal of the AAOS Web site of “improving the communication between orthopaedic surgeons and their patients” [24] and positively influence the health outcome of our patients.


We acknowledge the contributions of Caixia Zhao, MD, and Emily McClemens, PA-C, in preparation of this manuscript.


1. Albright J, Guzman C, Acebo P, Paiva D, Faulkner M, Swanson J. Readability of patient education materials: implications for clinical practice. Appl Nurs Res 1996;9:139-143. doi:10.1016/S0897-1897(96)80254-0.
2. Alexa® the Web Information Company. Available at: Accessed June 4, 2007.
3. American Academy of Orthopaedic Surgeons. Your orthopaedic connection. Available at: Accessed February 6, 2007.
4. Aslam N, Bowyer D, Wainwright A, Theologis T, Benson M. Evaluation of Internet use by paediatric orthopaedic outpatients and the quality of information available. J Pediatr Orthop B 2005;14:129-133.
5. Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America Web sites. J Bone Joint Surg Am 2008;90:199-204. doi:10.2106/JBJS.G.00347.
6. Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health 1997;87:1027-1030. doi:10.2105/AJPH.87.6.1027.
7. Bass L. Health literacy: implications for teaching the adult patient. J Infus Nurs 2005;28:15-22. doi:10.1097/00129804-200501000-00002.
8. Beall MS 3rd, Beall MS Jr, Greenfield ML, Biermann JS. Patient Internet use in a community outpatient orthopaedic practice. Iowa Orthop J 2002;22:103-107.
9. Beall MS 3rd, Golladay GJ, Greenfield ML, Hensinger RN, Biermann JS. Use of the Internet by pediatric orthopaedic outpatients. J Pediatr Orthop 2002;22:261-264. doi:10.1097/00004694-200203000-00026.
10. Berkman ND, DeWalt DA, Pignone MP, Sheridan SL, Lohr KN, Lux L, Sutton SF, Swinson T, Bonito AJ. Literacy, Health Outcomes: Evidence Report. Rockville MD: Agency for Healthcare Research and Quality/US Department of Health and Human Services; 2004.
11. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Munoz JA, Puyol JA, Lara M, Watkins KE, Yang H, McGlynn EA. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285:2612-2621. doi:10.1001/jama.285.20.2612.
12. Boulos MN. British internet-derived patient information on diabetes mellitus: is it readable? Diabetes Technol Ther 2005;7:528-535. doi:10.1089/dia.2005.7.528.
13. Brooks BA. Using the Internet for patient education. Orthop Nurs 2001;20:69-77. doi:10.1097/00006416-200109000-00011.
14. Cooley ME, Moriarty H, Berger MS, Selm-Orr D, Coyle B, Short T. Patient literacy and the readability of written cancer educational materials. Oncol Nurs Forum 1995;22:1345-1351.
15. Cotugna N, Vickery CE, Carpenter-Haefele KM. Evaluation of literacy level of patient education pages in health-related journals. J Community Health 2005;30:213-219. doi:10.1007/s10900-004-1959-x.
16. D'Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med 2001;155:807-812.
17. Davis TC, Mayeaux EJ, Fredrickson D, Bocchini JA Jr, Jackson RH, Murphy PW. Reading ability of parents compared with reading level of pediatric patient education materials. Pediatrics 1994;93:460-468.
18. Doak CC, Doak LG, Root JH. Teaching Patients With Low Literacy Skills. 2th ed. Philadelphia PA: JB Lippincott; 1996.
19. Flesch RF. How to Write Plain English: A Book for Lawyers, Consumers. 1th ed. New York NY: Barnes and Noble; 1981.
20. Fox S. Online Health Search 2006: Report. Washington, DC: Pew Internet and American Life Project; 2006.
21. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006;33:352-373. doi:10.1177/1090198105277329.
22. Friedman DB, Hoffman-Goetz L, Arocha JF. Readability of cancer information on the Internet. J Cancer Educ 2004;19:117-122. doi:10.1207/s15430154jce1902_13.
23. Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999;48:58-61.
24. Hirsch SA. Academy introduces Web-based medical education for fellows and patients. J Bone Joint Surg Am 2000;82:1665-1667.
25. Hoffmann T, McKenna K. Analysis of stroke patients' and carers' reading ability and the content and design of written materials: recommendations for improving written stroke information. Patient Educ Couns 2006;60:286-293. doi:10.1016/j.pec.2005.06.020.
26. Jackson RH, Davis TC, Bairnsfather LE, George RB, Crouch MA, Gault H. Patient reading ability: an overlooked problem in health care. South Med J 1991;84:1172-1175. doi:10.1097/00007611-199110000-00004.
27. Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel: Report. Millington, TN: Research Branch, Naval Technical Training Command; 1975.
28. Krempec J, Hall J, Biermann JS. Internet use by patients in orthopaedic surgery. Iowa Orthop J 2003;23:80-82.
29. Kutner M, Greenburg E, Jin Y, Paulsen C. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy. Washington, DC: National Center for Education Statistics; 2006.
30. Lewiecki EM, Rudolph LA, Kiebzak GM, Chavez JR, Thorpe BM. Assessment of osteoporosis-website quality. Osteoporosis Int 2006;17:741-752. doi:10.1007/s00198-005-0042-5.
31. Ley P, Florio T. The use of readability formulas in health care. Psychol Health Med 1996;1:7-28. doi:10.1080/13548509608400003.
32. Mosenthal PB, Kirsch IS. A new measure of assessing document complexity: the PMOSE/IKIRSCH Document Readability Formula. J Adolesc Adult Lit 1998;41:638-657.
33. Munro BH. Correlation. Statistical Methods for Health Care Research. 5th ed. Philadelphia, PA: Lippincott-Raven; 2005:239-258.
34. Murero M, D'Ancona G, Karamanoukian H. Use of the Internet by patients before and after cardiac surgery: telephone survey. J Med Internet Res 2001;3:E27. doi:10.2196/jmir.3.3.e27.
35. National Institutes of Health. MedlinePlus: How to write easy to read health materials. Available at: Accessed May 1, 2007.
36. Nielsen-Bohlman L, Panzer AM, Kindig DA. Health Literacy: A Prescription to End Confusion. Washington DC: Institute of Medicine, National Academies Press; 2004.
37. Wallace LS, Turner LW, Ballard JE, Keenum AJ, Weiss BD. Evaluation of Web-based osteoporosis educational materials. J Womens Health 2005;14:936-945. doi:10.1089/jwh.2005.14.936.
38. Weiss BD. Health Literacy: A Manual for Clinicians. Chicago IL: American Medical Association, American Medical Foundation; 2003.
© 2008 Lippincott Williams & Wilkins, Inc.