Methodology transparency is a core principle of high-quality clinical practice guideline development. Transparent, high-quality methods applied rigorously yield reproducible results and instill confidence in the user that the recommendations are made with the highest medical evidentiary support. Thus, transparency is a central value underlying criteria and methods published by multiple bodies which review quality of guidelines, including the US Institute of Medicine (IOM),1 and the international GRADE Working Group.2
The American College of Occupational and Environmental Medicine (ACOEM) has published transparent, evidence-based practice guidelines for the treatment of common health disorders among workers since 1997.3,4 The continually updated methods used for evidence search, critical appraisal, synthesis, and recommendation development were documented in the introduction to each edition and in other publications.5,6 The ACOEM Guideline Methodology Committee published an updated methodology in 2008,7 which ACOEM has used in publishing newer guidelines.8,9
Since the 2008 update, a number of advances in creating quality guidelines have been developed and are embodied in IOM's Clinical Practice Guidelines We Can Trust,1 and a series of publications from the GRADE Working Group.2 Tools and standards for the assessment of clinical practice guidelines (AGREE II10) and systematic reviews (AMSTAR11 and IOM1) have also been published. The updated ACOEM methodology, described fully in Methodology for ACOEM's Occupational Medicine Practice Guidelines—2017 Revision,12 is summarized herein and includes these advances in systematic review methods and guideline development. Summary tables comparing ACOEM's methodology to the standards described by AMSTAR, GRADE, AGREE II, and IOM are available in the complete methodology document.12
PURPOSE AND SCOPE
The purpose of the ACOEM Occupational Medicine Practice Guidelines (Guidelines) is to improve the quality of occupational medical care and disability management through identification of evidence-based best practices for key areas of occupational health care.10,13 The primary goals of the Guidelines are to suggest treatment most helpful in aiding recovery and restoring function after an illness or injury and to help focus delivery of the most effective treatment for a given condition at the earliest feasible point. Additionally, the Guidelines present information to improve the efficiency and accuracy of the diagnostic process, identify effectiveness and risks of treatment, and enhance patient autonomy by providing information needed for shared decision making.14
The Guidelines address the key domains of occupational health care practice including prevention, health promotion, diagnosis, causation determination, illness and injury treatment, and disability management. Examples of broad clinical questions that the Guidelines may address in these areas are listed in Table 1.10
The Guidelines include foundation guidelines covering the basic clinical assessment processes used, as well as guidelines for specific occupationally-related conditions and treatments, for example, for musculoskeletal, respiratory disorders, opioids, eye conditions, and traumatic brain injury. Clinical questions used to direct the evidence search and assessment are framed in a systematized format covering population of interest, intervention comparison and outcome (PICO).15 The questions for etiology, diagnosis, and prognosis are framed in a modified PICO format, emphasizing such factors as specific exposures or trauma for etiology or prevention, natural history for prognosis, and the reproducibility and performance of diagnostic tests accepted reference standards for clinical assessment.
ACOEM Guidelines are meant to apply to working-age adults (age 18 to 65).10 Although the primary focus is on injuries and illnesses that may have been caused by work, the Guidelines include common topics that affect the ability to work but are unlikely to be work-related. As many workers are now older than 65, guidance has been expanded to include all workers. As a practical matter, many studies include older adults and are incorporated in the evidence-base unless clear rationale for exclusion is found. This results in guidelines that may have substantially wider applicability than the target population.
Primary target users of the ACOEM Guidelines are physicians and other health care providers, and the health care organizations in which they deliver care.10,13 As workers’ compensation is an insurance system and a legal framework, the following are also members of the intended audience: patients, clinical case managers, insurers, third-party administrators, insurance claims managers, utilization reviewers, attorneys, judges, regulators, and policy makers.
The ACOEM Board of Directors appoints one physician to chair the guidelines development processes, act as the Guidelines editor-in-chief, and chair the Evidence-based Practice Committee (EBPC).10 A summary of the function, responsibility, and objective statements for the committees and panels involved with ACOEM's guideline-related activities follows:
The EBPC is comprised of the chairs of the Evidence-based Practice Panels (EBPPs or Panels). EBPC meeting may be attended by others who have been involved with previous ACOEM Guideline activities (eg, panel members and similar individuals). The EBPC is charged with coordinating updates of the ACOEM Guidelines, and also assists with identification of additional guideline topics and clinical questions that should be considered.
Multidisciplinary EBPPs are distinct panels of experts for each body part, system, or skill area covered by the Guidelines.16 Currently, there are panels for asthma, interstitial lung diseases, low back, cervical/thoracic spine, hand/wrist/forearm, elbow, shoulder, hip/groin, knee, foot/ankle, eye, chronic pain, opioids, traumatic brain injury, mental health, and disability prevention and management. Panels have an appointed chair and may have a co-chair. Prospective panel members apply for membership or are invited based on expertise, productivity, and/or prior performance. They are then screened for expertise and conflicts of interest (COI). Appropriate applicants are appointed and trained to develop and/or update evidence-based practice recommendations. Panels approve clinical questions to frame the literature search, review critical analyses of the literature based on this approved methodology, and develop, review and approve evidence-based recommendations for clinical practice, care management, and disability management. Panels are often subdivided into areas of practice or research interest at the discretion of the panel chair in discussion with the editor-in-chief (eg, medical management, therapy, chiropractic, interventional treatments, surgical care, psychological approaches) particularly when the panel has a large scope of work (eg, low back or chronic pain).
Research Team members are trained and draft preliminary clinical questions in PICO format for each guideline. The Research Team develops and documents search strategies and methods for each guideline topic and then conducts exhaustive systematic literature searches for each guideline topic and summarize studies in evidence tables. The Research Team then critically appraises, grades, and critiques each study, and finally drafts background text, rationale statements, and recommendations for each guideline topic and forwards these to the EBPP.
The Guideline Methodology Committee (GMC) developed the initial methodology for production and revision of the Guidelines and other evidence-based products. On an ongoing basis, the GMC refines, clarifies, and updates the methodology based on state-of-the-art internationally accepted methods. To ensure transparency, it publishes documents that describe and explain the methodology used for ACOEM evidence-based materials and products.7 The GMC ensures adherence to state-of-the-art methods by assigning methodologists to each panel and also approving panel members after reviewing applications, curriculum vitae, and COI information from interested individuals. It then trains panels in this methodology and guideline development process.17–20
PROCESS FOR DEVELOPMENT AND REVISION OF THE GUIDELINES AND OTHER ACOEM EVIDENCE-BASED PRODUCTS
The process for development of ACOEM Guidelines and evidence-based products was the work of the GMC and includes participation of the EBPC, review and formulation of recommendations by the Panels, stakeholder input, external peer review, and review by the ACOEM Board. Members of the guideline development groups are selected from applications of ACOEM members and nominees from relevant interest groups and professional organizations. All panel members are required to complete an application and a questionnaire to: (i) outline qualifications and interests; (ii) disclose potential COI; and (iii) agree to confidentiality procedures. Summaries of disclosures for all panel members are available online. All members of guideline development groups are required to complete training in ACOEM's evidence-based medicine methodology.7,12
Prioritization of Topics for Review and Recommendation
The editor-in-chief and Research Team in collaboration with each panel chair, works with the panels to identify clinical questions about important, useful, common, expensive, and/or controversial work-related diagnoses, tests, and procedures.17 Topics may be forwarded from panel members, peer reviewers, Board members, external stakeholders, and others. The following procedures are followed:
- Research Team identifies the most common occupational health problems, tests, and treatments in terms of frequency, cost, time off work, apparent benefits/harms, and rapid increases in utilization.
- Diagnoses are grouped into homogeneous diagnostic groups. Tests and treatments are identified as groups, if similar or synonymous, or individually if the criteria for use and evidence of effectiveness are likely to be relatively unique.
- Panels solicit suggestions on areas to include/examine, including identification of commonly used, and/or emerging diagnoses, work linkages, and commonly used (but not necessarily safe or effective) tests, medications or procedures from panel members and other stakeholders.
- PICO questions are finalized for a given guideline by the panel.
REVIEW AND FORMULATION OF RECOMMENDATIONS BY THE EVIDENCE-BASED PRACTICE PANELS
Panels with assistance from the Research Team, formulate recommendations for guidelines as follows:
Literature Evaluation: Literature Search and Study Selection
The Research Team conducts exhaustive systematic literature reviews for each guideline topic, using multiple electronic search services.10 Relevant citations in articles are reviewed. In order to identify all high- and moderate-quality original research studies, the literature search is broad and comprehensive, although limited to English publications. Animal studies are excluded other than for treatment of chemical ocular injuries for which there are few human studies. When searching the US National Library of Medicine database, Medical Subject Headings (MeSH) are used to identify studies relevant to tests, treatments, and diagnoses in question. A combination of MeSH and other terms is used to determine the method that will yield the most relevant studies. Gray literature is included primarily through searches of Google Scholar to identify publication biases. Ongoing literature surveillance is also used to assure currency of guidelines recommendations, as well as to provide literature to be incorporated during the next comprehensive update.
Treatment-Related Study Searches
For treatment-related study searches, randomized controlled trials (RCTs), and randomized crossover trials, quality guidelines, meta-analyses, and systematic reviews are the primary foci of these comprehensive, exhaustive literature searches.21 Prospective and retrospective cohort studies are searched if there are no RCTs or systematic reviews identified. High-quality guidelines, meta-analyses, and systematic reviews are sought primarily for verification of search completeness; they are independently assessed for reproducibility of conclusions. When there is a discrepancy identified, re-review of a topic is conducted to verify conclusions. Both evidence of effectiveness (ie, comparisons with placebo, sham, or control of known level of efficacy) and comparative effectiveness (ie, comparisons between two or more active treatments) are sought.
RCTs and randomized crossover trials are all selected for critical appraisal and quality grading. For evidence of harms, case reports, case series, retrospective cohort studies, and arms of RCTs are sought. For risk factor assessments, prospective and retrospective cohort studies are preferentially sought, with case-control or cross-sectional studies selected where cohort studies are absent. In some cases, studies with lower grades of evidence may be selected to examine current practice patterns or for other reasons. To ensure that all relevant, higher-quality studies are identified, the Research Team performs hand searches of reference lists in related articles.
Diagnostic or Screening Searches
For diagnostic study searches, all study design types are searched. Searches for these topics primarily focus on large, comparative trials looking at two or more diagnostic tests that are being compared. Ideally, one is the “gold standard” test for that condition. Key terms (eg, “Sensitivity and Specificity” [MeSH] OR “Predictive Value of Tests” [MeSH] OR “Gold-standard” OR accurate OR accuracy OR precision OR precise OR test) are used to identify the accuracy of the new test. Diagnostic studies are then summarized in evidence tables. Quality grading of diagnostic studies’ methods follows a different scheme than the grading scheme used for treatment. Highest scores are given to studies that compare the new test to a gold standard, if one exists. Timing of testing in relation to the progression of the disease state is also evaluated. Another criterion for judging the quality of studies of diagnostic methods is availability in the publication of data to determine or explicit documentation of specificity and sensitivity. Studies that compare a new test to an established gold standard are evaluated first. Studies that compare a new test to another test, but not a gold standard, are also evaluated. In order to ensure all relevant studies are included in the review, researchers also utilize literature identified, including from ongoing literature surveillance, stakeholders, panel members, and reference lists from previously identified studies.
Search Term Documentation
Search strategies and methods, including specific databases, search terms, number of studies found (eg, regarding treatment efficacy searches including RCTs and crossover trials) are documented. A search results section is included as a footnote for each evidence table. This section includes databases searched, limits on publication dates and languages, search terms used, number of studies found from all databases searched, total number of articles screened, number meeting inclusion criteria, number critically appraised, and total number of studies included. Tracking logs that document the search process, search terms, limitations, etc, are also published in order to maintain transparency.
The Research Team reviews the abstracts of all citations found in the bibliographic search and identifies studies relevant to the topic that might meet the inclusion criteria (eg, in English, RCTs that address treatment questions, relevant literature for adverse effects, and comparative studies for diagnostic or screening tests) as adequate evidence and that could be used as the basis for evidence-based guidance statements. Researchers then retrieve the full text of these articles and perform a second screening process of the study in order to determine which studies meet the inclusion criteria to be considered as adequate evidence for these purposes. For those studies accepted as providing adequate evidence, individual article quality ratings are included in the evidence tables.
Literature Evaluation: Critical Review of Studies
The Research Team reviews in detail each study that meets inclusion criteria and summarizes important information from each in an evidence table.10 Evidence tables include first author's last name, year of publication, study design, quality rating score, population sample, age, sex, treatment comparison, follow-up time, results, conclusions, and comments relevant to the study.22 Potential COI and study sponsorships are also recorded. The evidence presented in the tables is limited to primary studies. In most cases, quality systematic reviews, meta-analyses, and professional guidelines are reviewed for comparison and assessment of reproducibility. The relative ranking of study designs for theoretical robustness is included in the complete methodology document.12 Table 2 summarizes the level of confidence levels for the different study designs. While study design should confer various levels of confidence in the reproducibility of the results, how studies are conducted and analyzed is variable and must be specifically appraised.
The Research Team critically appraises, grades, and critiques each study. Reviewers grade each study using a numerical quality score (Tables B and C in complete methodology12). Excepting fatal flaws, studies, based on their scores, are designated as high-, moderate- or low-quality evidence and report the scoring in the combined quality assessment table (eg, scores greater than or equal to 4.0 or higher are designated moderate or high quality). The highest score for studies is 11 points.
After Research Team assistants complete the evidence tables, researchers with graduate degrees (Master, PhD, MD) score each study for quality. Studies are critiqued for methodological strengths and weaknesses and assessed for robustness and validity of the conclusions derived from presented data.23 Once the body of quality evidence is assembled, scored, and critiqued, it is graded. Draft recommendations are then formulated and sent to the panel. In all cases, a Research Team physician performs a secondary review for clinical relevance and logic. Panels perform additional quality review.
DEVELOPMENT OF GUIDELINES AND RECOMMENDATION STATEMENTS
Panels review and modify draft recommendations formulated by the Research Team.10 Panels (and/or sub-panels) review evidence tables and summaries, draft recommendations, and original studies if needed. After review, panels determine the strength of evidence ratings for each topic (Table D in complete methodology12) and finalize recommendations for all clinical questions. Table 3 illustrates the minimum thresholds used for evidence-based recommendations. If a sub-panel is employed, its recommendations are forwarded to the entire panel in aggregate for additional discussion. Each recommendation is reviewed, edited (if necessary), and labeled as “strongly recommended,” “moderately recommended,” “recommended,” “consensus-recommended,” “consensus-no recommendation,” “consensus—not recommended,” “not recommended,” “moderately not recommended,” and “strongly not recommended” (Table E in complete methodology12). Panel unanimity is nearly always achieved primarily through iterative drafts. Failing attainment of unanimity, consensus is sought for all recommendations and rationales in each guideline. When consensus is not possible, a vote is taken. Minority statements may be included.
The health benefits, adverse effects, risks, and relative costs of each recommended test or treatment are explicitly considered and discussed when formulating recommendations.10 Benefits should significantly exceed risks. Each recommendation is to specify to which condition it applies (Table 4).
As funding/sponsorship of pharmaceuticals and devices or appliances is almost universally commercial, since 2014, evidence tables have included information about potential COIs that are published. It is problematic that there are studies of expensive interventions conducted in clinical settings where there is significant bias to support the organization's clinical business; currently, there is no clear method to address this potentially significant source of funding bias. In certain areas, this may make little difference as comparisons were between a medication and placebo and results may be consistent and considerable. However, in other studies, comparison groups may have been sub-optimally treated (eg, low-dose of ibuprofen) and produced a bias in favor of the medication or device, the results never independently replicated, or the results conflict with independent studies. In addition, industry-sponsored studies have been shown to frequently have better results and lower complication rates than studies conducted by independent investigators.26–28
Studies that include the general adult population are necessary to develop most recommendations. However, consideration is given to the extent to which findings may or may not be applicable to employed populations. ACOEM's “First principles” (Table 5) of clinical logic and ethics should be observed in formulating guidelines and clinical recommendations.
Each Guidelines recommendation includes an explicit link with supporting evidence, an evidence table, and list of references.10 Each is accompanied by a paragraph that describes the panel's conclusion about the evidence found on that question, for example, the rationale for the specific recommendation. These paragraphs succinctly explain how in formulating the recommendations the panel interpreted and weighed the evidence and balanced evidence of effectiveness or accuracy against potential harms and relative cost-effectiveness, for example, if the quality of the synthesized evidence was inconsistent, the Panel may comment on how they interpreted and weighed the evidence in a logical and fair way and adhered to “first principles” (Table 5).10 Final recommendations are then drafted and approved (Table 4).
EXTERNAL PEER REVIEW
ACOEM conducts external peer review of the Guidelines to: (1) assure all relevant high-quality scientific literature related to the topics has been found; (2) assure that the important evidence from the scientific literature relevant to the Guidelines has been accurately interpreted; (3) solicit opinions on whether the findings and recommendation statements are appropriate and consistent with the evidence; and (4) obtain general information on the Guidelines’ conclusions and presentation from external topic experts.10 These experts may also review the methodology used as well as summaries of the critically appraised evidence and the recommendations in each area. Names of peer reviewers are listed, along with their affiliations, for those not desiring anonymity. Panels review external peer reviewer comments and make any final modifications to the Guidelines. In addition, for 2 weeks, a prepublication version of all guidelines is posted at the MDGuidelines ® site for public comment.
To understand the needs and preferences of individuals and organizations who use or are affected by the use of clinical practice guidelines in workplace settings and workers’ compensation system, ACOEM solicits ongoing input from stakeholders—clinicians, health care systems, labor representatives, workers/patients, employers, utilization reviewers, case managers, insurers/third-party administrators, attorneys, regulators, and policy makers—by inviting them to submit comments through a form on ACOEM's web site.10 ACOEM also seeks input on the clinical questions researched for each guideline.
The Guidelines are pilot tested by clinicians, utilization review managers, case managers, state workers’ compensation systems, etc, who comment on use of the Guidelines in their daily practice or management activities to determine if they are clear, easy to use, and useful.10,31 The Guidelines may be modified based on feedback from pilot testing if suggestions increase usability. In 2014, Reed Group, Ltd., the Guidelines publisher, conducted a pilot test and redesigned its web site to address input received during this process.
REVIEW BY THE GMC AND ACOEM BOARD OF DIRECTORS
During the evidence-based development process, a designated methodologist from the GMC works with the panels, editors, and Research Team to ensure this methodology is followed in the literature evaluation process and in development of conclusion, rationale, and recommendation statements. ACOEM Board members may comment on the guidelines during the external review period and these comments are reviewed by the Panel and included in a manner identical to other external peer reviewers. Panels and the Research Team have complete editorial independence from ACOEM and Reed Group, Ltd., neither party influences the Guidelines.
ACOEM reviews the literature periodically to identify any major changes in the evidence-base by content area.10 Subsequent updates of the Guidelines include a full review of previous recommendations. Comprehensive updates conducted every 3 to 5 years involve panel review of new evidence and revision of the recommendations. Major changes in literature necessitate focused updates.
APPLICABILITY/TOOLS FOR PUTTING IT INTO PRACTICE AND MONITORING/AUDITING CRITERIA
The ACOEM Guidelines are developed and maintained by ACOEM.10 ACOEM members, personnel, and contractors are responsible for research and writing and control quality of the Guidelines. The Guidelines are published through Reed Group's MDGuidelines ® (www.MDGuidelines.com). Reed Group, Ltd. has released a new tool, Diagnosis and Related Treatments (DART), which provides instant access to ACOEM diagnostic and treatment recommendations and the supporting evidence. ACOEM has also developed monitoring and auditing criteria for each guideline (see complete methodology document12).
In conclusion, ACOEM's rigorous, standardized process was developed to ensure the reproducibility and transparency of the guideline development process. This process and the criteria for identifying and assessing relevant scientific evidence from the published literature are based on generally accepted principles and methodologies of EBM and evidence-based practice guideline development which have been widely discussed and agreed upon internationally. Use of this methodology results in valid, consistent, logical, and robust recommendations for clinical practice that have the greatest potential to improve the health and function of workers and other populations.
The authors acknowledge the assistance of the ACOEM Research Team at the University of Utah: Emilee Eden, BS, Atim Effiong, MPH, Matthew T. Houston, Harkomal Kaur, Jenna K. Lindsey, Pranjal A. Muthe, Michael L. Northrup, Natalie V. Pratt, Brenden B. Ronna, BS, and Skyler D. Walker.
1. Institute of Medicine. Standards for Developing Trustworthy Clinical Practice Guidelines; 2911. Available at: http://http://www.nationalacademies.org
/hmd/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed June 12, 2017.
2. GRADE Working Group. Criteria for Applying or Using GRADE; 2016. Available at: http://http://www.gradeworkinggroup.org
/docs/Criteria_for_using_GRADE_2016-04-05.pdf. Accessed June 12, 2017.
3. Harris J, Blais B, Brigham C, Kuhnen A, Wolens D. OEM Health Information Press, Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery in Workers. Beverly Farms, Mass: 1997.
4. Glass L, Blais B, Genovese E, et al. Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery in Workers. OEM Health Information Press, 2nd ed.Beverly Farms, Mass: 2004.
5. Harris J. Harris J, Loeppke R. The ACOEM occupational medicine practice guidelines. Integrated Health Management: The Key Role of Occupational Medicine in Managed Care, Disability Management and Integrated Delivery Systems
. Beverly Farms, Mass: OEM Press; 1998. 179–183.
6. Harris J, Glass L, Mueller K, Genovese E. Committee APG. Evidence-based clinical occupational medicine: updating the ACOEM Occupational Medicine Practice Guidelines. Clin Occup Environ Med
2004; 4:341–360. viii.
7. Harris JS, Sinnott PL, Holland JP, et al. Methodology to update the practice recommendations in the American College of Occupational and Environmental Medicine's Occupational Medicine Practice Guidelines, second edition. J Occup Environ Med
8. Hegmann KT. Occupational Medicine Practice Guidelines. American College of Occupational and Environmental Medicine, 3rd ed.Elk Grove, IL: 2011.
9. Hegmann KT. American College of Occupational and Environmental Medicine, Occupational Medicine Practice Guidelines, 2008 Revision. Elk Grove, IL: 2008.
10. Brouwers M, Kho M, Browman G, et al. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J
11. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol
12. American College of Occupational and Environmental Medicine. Methodology for ACOEM's Occupational Medicine Practice Guidelines—2017 Revision; 2017. Available at: http://http://www.acoem.org
/uploadedFiles/Knowledge_Centers/Practice_Guidelines/ACOEM%20Practice%20Guidelines%20Methodology.pdf. Accessed June 12, 2017.
13. Riegelman RK. Studying a Study and Testing a Test: How to Read the Medical Evidence. 5th ed.Philadelphia, PA: Wolters Kluwer Health; 2005.
15. Verbeek J, van Dijk F. World Health Organisation, A Practical Guide for the Use of Research Information to Improve the Quality of Occupational Health Practice. Geneva, Switzerland: 2006.
16. Bigos S, Bowyer O, Braen G, Brown K, Deyo R, Haldeman S. Acute Low Back Problems in Adults. Clinical Practice Guideline No. 14. AHCPR Publication No. 95–0642. Rockville, MD: Agency for Health Care Policy and Research Public Health Service, US Department of Health and Human Services; 1994.
17. Scottish Intercollegiate Guidelines Network (SIGN). SIGN 50: A Guideline Developer's Handbook; 2004.
18. Fretheim A, Schünemann HJ, Oxman AD. Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst
19. Kunz R, Fretheim A, Cluzeau F, et al. Guideline group composition and group processes: article 3 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc
20. Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst
21. Egger M, Davey-Smith G, Altman D. Systematic Reviews in Health Care: Meta-Analysis in Context. Hoboken, NJ: John Wiley & Sons; 2008.
22. Holland J. Development of a clinical practice guideline for acute low back pain. Curr Opin Orthop
23. Greenhalgh T. How to Read a Paper: The Basics of Evidence-based Medicine. Hoboken, NJ: John Wiley & Sons; 2014.
24. Hoogeboom T, Oosting E, Vriezekolk J, et al. Therapeutic validity and effectiveness of preoperative exercise on functional recovery after joint replacement: a systematic review and meta-analysis. PLoS ONE
25. Zwarenstein M, Treweek S, Gagnier J, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ
26. Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA
27. Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine
28. Steinbrook R. Peer review and federal regulations. N Engl J Med
29. Bruckman RZHJ. Occupational medicine practice guidelines. Manag Care Occup Med
30. Haynes R, Sackett D, Guyatt G, Tugwell P. Clinical Epidemiology: How to do Clinical Practice Research. 3rd ed.Baltimore, MD: Lippincott Williams & Wilkins; 2005.
31. Harris JS, Glass LS, Ossler C, Low P. Committee APGDPS. Evidence-based design: The ACOEM practice guidelines dissemination project. J Occup Environ Med