Hafner, Brian J. PhD; Geil, Mark D. PhD
Clinical decision making can be informed through evidence-based practice (EBP), an approach to clinical care in which a practitioner's training and experience are combined with evidence established through scientific research. Sackett et al.1 define EBP as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients,” noting that EBP should be the integration of “individual clinical expertise with the best available external clinical evidence from systematic research.” Thus, EBP encourages the review and synthesis of available and appropriate empirical evidence into clinical practice to promote effective and judicious patient care. Although relatively new to the orthotic and prosthetic (O&P) profession, EBP is well established as a valuable component of care in related healthcare fields such as physical2 and occupational therapy.3
Few would argue that certain topics in O&P have more developed bodies of empirical evidence than others. Yet among those areas that are relatively well supported by a body of literature, there still often exists a diversity of clinical and scientific opinion. This may be due to a relative unfamiliarity with the body of knowledge, limited access to the evidence, or conflicting results within the existing literature. Such topics lend themselves to thorough and systematic review of the evidence and discussion among subject matter experts so as to establish and document the current state of the science. Such processes are known as consensus4 or state-of-the-science5 conferences in related professions and play a critical role in establishing a scientific foundation for EBP.
The American Academy of Orthotists and Prosthetists (AAOP), under a 5-year grant sponsored by the United States Department of Education, has facilitated and supported nine State-of-the-Science Conferences (SSCs) between 2003 and 2009 on specific topics related to the provision of O&P care. The goal of the AAOP SSC program was to evaluate the available scientific information on aspects of O&P care and to develop statements that advance understanding of those specific issues that will be useful to health professionals and the public.
This growing corpus of SSC results has become an important component of EBP available to clinicians; therefore, widespread dissemination of these results has been sought. SSC Official Findings have been published and mailed as supplements to the Journal of Prosthetics and Orthotics to journal subscribers and AAOP members. Each of the first eight SSC findings was mailed to an average of 2,661 people. The SSC findings have also been posted online at the official websites of the AAOP6 and Lippincott Williams & Wilkins,7 the publisher of the Journal of Prosthetics and Orthotics. In addition, the findings of each conference have been converted to an online course and posted to the AAOP's Paul E. Leimkuehler Online Learning Center (OLC).8 Moreover, content from selected SSCs was made available to the AAOP membership in the form of clinical posters. A total of 15,185 clinical posters have been distributed to AAOP members. Although data are available related to how many official findings and posters have been mailed and how many times different online content has been assessed, no study has yet been performed to assess the dissemination of SSC results and the impact on the clinical community.
The purpose of this research was to survey representatives of O&P community and to 1) assess their awareness of the SSC official findings, 2) determine the impact this information has had on clinical practice, 3) document barriers to dissemination, and 4) suggest changes to the SSC program to improve the potential for achieving the program objectives in the future.
A custom, 15-item survey was designed by the investigators to solicit individuals' awareness of and experience with the information disseminated as a part of the AAOP SSC program. Thirteen survey items were presented with mutually exclusive categorical (i.e., dichotomous or ordinal) responses. Two items were presented as open-ended questions that allowed the respondent to provide additional detail regarding the provided response. The survey required approximately 5 minutes to complete.
Survey items were grouped into three categories, including demographics (five questions), awareness (five questions), consumption (four questions), and application (one question). Demographic questions surveyed the age, years of clinical practice, clinical certification status (i.e., Certified Orthotist, Certified Prosthetist, or both), highest level of academic education, and number of practitioners in the office in which the respondents practiced. Awareness questions polled the respondents' cognizance of the SSC program, identification of the purpose of the program, and recollection of the number of conferences and names of specific conferences (both from memory and after a list was provided). Consumption questions appraised the number of SSC Proceedings received, the number of proceedings read (both in whole and in part), and the number of online courses taken by the respondents. The final open-ended question surveyed if (and how) information gained from the SSC program had been used to change the respondents' clinical practice. No personal data were collected. Approval for study procedures was obtained from the University of Washington Internal Review Board (HSD No. 356500).
SURVEY ADMINISTRATION AND ANALYSIS
The survey was administered through the WebQ open-source survey software (Figure 1) hosted at the University of Washington. An invitation to participate in the survey was sent to 2,179 members of the AAOP mailing list. Three additional email reminders were sent before the survey was closed. The academy provided a “click through” link on the survey invitation that allowed the investigators to tabulate the number of individuals who visited the survey through this announcement mechanism. Solicitations to participate were also posted to the O&P LISTSERV on the same day when the original email and reminder emails were sent. The O&P Listserv includes approximately 4,500 individuals, many of whom are also on the AAOP mailing list. Given the redundancy on these lists, the exact number of individual solicitations that was disseminated is unknown, but it is between 4,500 and 6,679.
On conclusion of the survey period, survey records were exported from WebQ to Microsoft Excel 2003 (Microsoft, Redmond, WA), and SPSS version 17.0 (SPSS Inc, Chicago, IL) was used for data visualization and analysis. Response distributions for subject demographics were visually inspected in Excel. Expected associations among ordinal demographic responses, such as age and years of clinical practice, were assessed with a Spearman rank (rS) correlation coefficient using SPSS. Similarly, hypothesized associations between an awareness of the SSC program and consumption of the SSC materials or ordinal demographics were assessed with a Pearson Chi Square (χ2) analysis. If significant associations were detected among multiple response categories, the adjusted residuals were compared with the critical value in a post hoc analysis to assess the significance of individual categories. Significant findings were noted as those that achieved a p value less than or equal to 0.05.
ONLINE LEARNING CENTER ANALYSIS
At the time the SSC dissemination survey was administered, the Paul E. Leimkuehler OLC included courses based on the results of the eight SSCs conducted to date. Content from SSC courses is freely available to practitioners and students. Examinations covering the course material are freely available to students as a part of their curricula and for a fee to practitioners seeking continuing education credit. To expand on the survey question related to dissemination of SSC information through the OLC mechanism, the investigators requested and were provided with access to the SSC-related OLC results.
The academy provided the investigators with access to the OLC database. This included information such as the total number of course users, examination attempts, examination pass rate, average examination scores, and responses to the optional, postexamination course evaluations. The totals were also separated into students and practitioners cohorts for group analysis. These data were exported by the investigators into Microsoft Excel for visualization and analysis.
The online dissemination survey was made available between June 12, 2009 and June 29, 2009. At the time of the survey, eight SSCs had been conducted, and an online professional continuing education course was available for each. A total of 507 respondents completed the survey. The AAOP “click-through” data showed that 215 individuals initially visited the survey from the original, AAOP-generated email announcement. The three email reminders solicited 161, 117, and 82 additional views, for a total of 575 views from the AAOP-generated email. The exact number of views from the O&P Listserv is unknown.
The age of the respondents presented as a negatively skewed distribution with a median age between 51 and 55 years of age (Figure 2). Respondents' years of clinical experience varied from less than 5 years to more than 51 years (Figure 3). Approximately half of all respondents (46.5%) fell into three distinct, nonconsecutive, categories (i.e., 0-5 years, 11-15 years, and 26-30 years). As expected, years of clinical experience correlated strongly (rS = 0.80, p < 0.01) with respondent age. The majority (82.4%) of respondents were certified practitioners. A total of 40.6% of respondents were certified as a prosthetist-orthotist, 19.4% as a prosthetist, and 22.4% as an orthotist. Residents accounted for 6.5% and students for 4.4% of the total responses. An increasing level of clinical certification (i.e., student, resident, single certification, and dual certification) was weakly correlated with increasing age (rS = 0.23, p < 0.01). Respondents' highest level of education was evenly split between bachelors degrees (35.0%) and certificates (35.8%). A total of 18.0% of respondents had obtained a master's degree, and 2.8% possessed a doctorate degree. No correlation (rS = −0.04) was detected between age and increasing level of education. Most respondents (59.1%) practiced in small clinical offices (i.e., between one and four persons), whereas few (11.1%) worked in larger offices with eight or more practitioners (Figure 4).
The majority (72.4%) of respondents indicated that they were aware that the academy had conducted SSC on O&P-related topics in the period of time defined by the survey (2003-2007). The results described throughout the remainder of this article are based on those respondents who indicated that they were aware of the SSC program (i.e., 368 of the 507 total respondents). Awareness was greatest among those with a master's or doctorate degree, residents or those certified as both a prosthetist and orthotist, and those working in offices with seven or more practitioners (Table 1). However, awareness in these subgroups was not significantly different (p > 0.05) than the awareness reported by all respondents. Respondents typically recalled fewer conferences that were held between 2003 and 2007 (i.e., less than eight conferences; Figure 5). Only 15.5% of respondents correctly identified the “7 to 8” category. Further, 3.9% of respondents indicated that there had been nine or more SSCs held. The majority of all respondents correctly identified the primary purposes of the SSC program (Figure 6). Of the 10 available options (i.e., 5 correct and 5 incorrect), more than 56.8% of all respondents accurately identified those that were correct. Only one incorrect option, “establish best practices for performing clinical care,” was noted as a purpose of the SSCs by the majority (63.5%) of respondents. All other incorrect answers were noted by a minority of all respondents. Although most respondents were aware of the SSC program, only 38.2% were able to recall the correct title (or subject) of at least one SSC, by recall alone. Once provided with the titles, only 9.3% noted that they were unfamiliar with none of the SSC topics. Awareness of specific conferences or topics was highest for those conferences held first and lowest for those held most recently (Figure 7). Awareness of each SSC was significantly higher (p < 0.01) in those representing the clinical discipline (i.e., prosthetics or orthotics) to which the SSC was related (Table 2). For example, orthotists were significantly more likely (p < 0.00) to be aware of the scoliosis SSC than those with only prosthetics training.
Most (83.8%) respondents indicated that they had received one or more of the Official SSC Proceedings, a supplement provided along with the quarterly issue of the Journal of Prosthetics and Orthotics to all academy members. However, only 28.4% noted that they had received all eight publications. A large majority (72.7%) of respondents indicated that they had read at least one SSC Proceedings cover-to-cover, and even greater percentage (84.9%) indicated that they read a part of at least one (Figure 8). Very few (3.9%) of respondents indicated that they had read all eight SSC Proceedings, but a larger proportion (12.2%) noted that they had read at least a part of all eight. Of those 342 individuals who reported having received at least one SSC Proceedings, 92.6% reported reading at least part of one, and 85.3% reported reading one or more cover-to-cover (Table 1). In general, choosing to read the proceedings was greatest for those with a master's or doctorate degree, students and residents, and those in larger offices. However, none of these demographic characteristics were significantly different than consumption by the overall group. A small percentage (20.6%) of respondents indicated that they had used the Paul E. Leimkuehler OLC to take at least one SSC course. Survey-reported participation in individual SSC courses through the OLC varied from 5.5% to 10.5% of all respondents, with “Prosthetic Foot/Ankle Mechanisms” being the most-attended course. Notably, orthotic courses were most attended by those with orthotics clinical training, and prosthetic courses were similarly attended by those with prosthetic training. This difference was statistically significant for the “Orthotic Treatment of Idiopathic Scoliosis and Scheuermann's Kyphosis” (p < 0.02) and “Orthotic Treatment of Deformational Plagiocephaly, Brachycephaly and Scaphocephaly” (p < 0.04) courses.
Data extracted from the academy-provided database showed that 1,070 users had accessed the SSC content through the OLC at the time of the survey (i.e., June 2009). The “Orthotic Treatment of Idiopathic Scoliosis and Scheuermann's Kyphosis” course was the most accessed (196 users), and “Outcomes Measures in Lower Limb Prosthetics” was the least accessed (42 users). The ratio of practitioners-to-students varied across the conferences, ranging from a high of 1.63 (i.e., for “Orthotic Treatment of Deformational Plagiocephaly, Brachycephaly and Scaphocephaly”) to a low of 0.36 (i.e., “Post-Operative Management of Lower Limb Amputations”; Figure 9). In general, those who rated the OLC courses found them to be of “good” to “excellent” clinical relevance and educational quality. Most indicated that the information gained would influence their clinical decisions in the future (Table 3).
A response to the final survey question, which polled the impact of the SSC program to clinical practice, was provided by 20.5% of respondents. Responses predictably varied, but several common themes for impact of the SSC program emerged. Respondents indicated that the SSC Proceedings most commonly impacted clinical practice by providing objective evidence to substantiate clinical decisions, presenting alternative treatment options, and furnishing clinicians with general education on the SSC topics. Multiple respondents also indicated that the SSC materials were useful for discussions with patients and physicians, that they gave the practitioners more confidence in their clinical decisions, and helped them to select better outcomes to use in assessing their own patients. Each conference was specifically noted by at least one respondent as having altered their clinical practice in some way.
SSCs can be valuable resources for developing evidence to support O&P clinical practice. By design, they involve thorough and systematic review of the scientific literature related to a specific clinical topic and subsequent analysis by subject matter experts. However, SSCs cannot inform clinical practice if their results are not appropriately disseminated and consumed. The purpose of this study was to provide the first investigation on SSC awareness in the O&P clinical community in an effort to identify those barriers that might exist and thus hinder future SSC dissemination.
The survey instrument used as the primary basis of this investigation was developed by the authors and was not independently tested for validity or reliability. For the purposes of this study, thorough determinations of validity and reliability were considered impractical. Given that many of the questions involved simple numerical recall or solicited commonly surveyed demographic information, no major validity concerns were perceived.
Results of a survey of this nature are highly dependent on respondent demographics. These data (Figures 2–4) were considered representative of the industry at large and compared favorably with the demographics of respondents in a survey of 2,578 certified practitioners with 718 completed responses conducted by the American Board for Certification (ABC) in Orthotics, Prosthetics, and Pedorthics.9 The largest subset of respondent ages in the ABC study was 45 to 54 years, similar to this study. Regarding practice experience, most ABC respondents reported 10 to 19 years of practice experience, but the categories reported were less discriminatory than in this study. Furthermore, the data reported here included responses from students, perhaps explaining the proportion (i.e., 18.5%) of respondents reporting 0 to 5 years of experience. Finally, this study was similar to the ABC survey in facility size. Both reported a majority of respondents working in a facility with five or fewer practitioners. The agreement between these surveys' demographics suggests that the present survey reached a representative sample of O&P practitioners.
In general, the survey revealed a high level of general and specific awareness of SSCs, with 72% of respondents indicating general awareness and 46% of respondents attempting to list the topic of one of more SSCs. The survey assessed recall of specific SSC topics in two ways. The first and potentially more challenging method was to solicit recall of the number of SSCs held by the academy and recall of specific SSC topics with no prompting. The second was to assess recall of specific topics after seeing a list of those conference topics that were the focus of the SSCs. Most of the 233 respondents who attempted to list SSC topics without a provided list did indeed recall some or all of the 8 topics accurately. Not surprisingly, many respondents incorrectly recalled other AAOP courses that were not SSCs, including topics such as EBP and single subject research. It should be noted that participants were not given specific instructions regarding recall from memory alone, and some may have looked up SSC titles from their bookshelves or the internet.
Once participants were provided a list of topics, self-reported awareness was also high. Only 9.3% still did not recall the existence of any of the listed SSCs (Figure 7). Considering the survey was distributed to a worldwide audience through the O&P LISTSERV, some lack of awareness may be expected and could explain this respondent group's unfamiliarity with the SSC program. There is an interesting distribution in topic awareness, but no statistical significance between topics. The oldest conference was recalled the most, as might be expected. Thereafter, awareness of subsequent SSCs declined, with two exceptions: a 2005 conference on Prosthetic Foot/Ankle Mechanisms and a 2009 conference, the most recent conference at the time of the survey, on the Biomechanics of Ambulation after Partial Foot Amputation. An increased awareness of the most recent course is not unexpected. However, reasons for the slightly greater recall of the Prosthetic Foot/Ankle Mechanisms course are not obvious. Results from online course access may help to explain the increased awareness. Combining data from students and practitioners, the top two most utilized on-line courses were Orthotic Treatment of Idiopathic Scoliosis and Scheuermann's Kyphosis (also the most-recalled topic) and Prosthetic Foot/Ankle Mechanisms. This suggests that the topic of prosthetic foot/ankle mechanisms was particularly relevant to a broad number of practitioners.
The reversed chronological trend in awareness (i.e., oldest topics were recalled the most) is not immediately intuitive. The older SSC topics have certainly existed longer and may thus have had more time to be discussed and applied. However, those conferences were also held before 18.5% of respondents entered clinical practice (i.e., those who had practiced fewer than 5 years). Furthermore, older material typically has an inherent “shelf-life,” both in applicability to practice and in recall among practitioners. The fact that the oldest topics were generally recalled the most suggests that SSCs have had a considerable impact and a prolonged period of relevance.
Awareness is an important measure of dissemination, but more critical indicators may be consumption and application. In these domains, SSCs can be considered well used and relevant to practice. Physical dissemination methods affect both awareness and consumption, and these results suggest that dissemination through physical print media is effective. More than 92% of survey respondents who recalled receiving a print copy of at least one SSC official findings reported reading at least part of one, and more than 85% had read at least one cover-to-cover. Numbers of respondents who reported reading multiple (or even all) SSCs were smaller. This result was expected because practitioners who specialize within a certain discipline will find certain SSC topics to be more relevant to their practice.
This study did not attempt to distinguish between consumption of print versus digital or electronic findings, but some individual answers suggested modest online consumption. For example, 42 participants reported that they had read, in whole or in part, more SSC findings than they reported having received, indicating they either read physical copies that were not sent to them or they read the findings online. That the findings were read and used are considered by the authors to be of greater importance than the format in which they were obtained.
Data from the AAOP OLC regarding access frequency of virtual continuing education courses associated with each SSC may be less subjective than self-reported recalls of awareness and application. Access by students suggests that SSCs are useful not only for EBP but also for education. Online course utilization by topic did not match trends in awareness. As opposed to recall and awareness, the online courses that have been available the longest should be expected to have the most access. This was not the case, however, indicating that topic is more important in the case of online learning.
Application of SSC evidence is more difficult to assess. The final survey question asked: “Has the information from any of the SSCs changed your clinical practice? If so, please indicate which conference and how it changed your practice in the box below.” More than one hundred people responded to this question. A subjective analysis of those responses showed that approximately one third were able to note a specific change to clinical practice that resulted from one or more of the SSCs. For example, one respondent stated, “I revisited the foot ankle mechanisms and decided to venture out from the constants I've been using. It's been enlightening.” Approximately one half of the respondents to this question reported that one or more SSCs produced nonspecific change in practice or informed practice in some way, and approximately one tenth suggested that SSCs had confirmed their current practices but did not necessarily promote a change in them. For example, a respondent stated, “I don't know that it “changed” my clinical practice, but it gave me some evidence for discussions with patients and other health care providers.” Although the goal of this project was not to determine which topics were most widely applied, responses to the application question did reveal three commonly mentioned SSCs: Biomechanics of Ambulation after Partial Foot Amputation; Orthotic Treatment of Deformational Plagiocephaly, Brachycephaly, and Scaphocephaly; and Outcome Measures in Lower Limb Prosthetics. The fact that the latter SSC topic was noted to be particularly clinically relevant, and yet, the associated online course is the least used by a substantial margin is difficult to explain. The online course for the prosthetic outcomes was, like all the online courses, viewed favorably in user feedback collected by the OLC. It received high marks for clinical relevance, overall effectiveness, and likelihood that users will apply the material and complete another online course in the future. That traditional outcome measures are not currently required as a component of clinical care in O&P may be one reason that the OLC course is not commonly used. However, as EBP gains prominence in O&P, outcomes may become more commonly used. If so, the content provided by this SSC (and others like it) may see an increased importance in coming years.
The results of this investigation indicate that students and practitioners are not only generally aware of the existence of the AAOP's SSCs but that they also read the SSC Proceedings, access the material in online courses, and apply the material in clinical practice. These findings demonstrate that SSCs can be a valuable component of EBP in prosthetics and orthotics and that future dissemination efforts are expected to have similar value. Furthermore, results of practical application of SSC information indicate a growing acceptance of EBP in O&P.
The authors would like to thank the American Academy of Orthotists and Prosthetists for disseminating the survey announcement through their email distribution list and for providing information related to use of the online learning center.