The strength of systematic reviews relies inherently on the methodology of the included studies, which inevitably to attain a high level of evidence must be reliable and transparent. We read with great interest a study conducted by Momeni et al.,1 in which the authors evaluated the methodologic quality of systematic reviews in the hand surgery literature using the validated critical appraisal tool AMSTAR.2 The authors concluded that the median AMSTAR overall score of systematic reviews in the hand surgery literature was 7 of 11, which corresponds to a fair to good score. However, we noted that the study has two major underlying issues. First, the authors stated that the majority of the evaluated systematic reviews provided only a list of included studies, but none provided a list of excluded studies. The fifth criterion in the AMSTAR tool states that fulfillment of this criterion requires “a list of included and excluded studies should be provided.”2 However, the authors stated that providing solely a list of included studies was sufficient to satisfy this AMSTAR criterion. Modifying the original criterion of AMSTAR can potentially negatively influence the validity of the AMSTAR tool and thus lead to misleading results. Second, the last criterion of the AMSTAR tool, which consists of evaluating conflict of interest, was reported in 81 percent of the included systematic reviews. This criterion states that “potential sources of support should be clearly acknowledged in both the systematic review and the included studies.”2 However, the authors consider acknowledgment of source of support in systematic reviews only and not the included studies. Popovich et al. found that more than half of Cochrane reviews did not assess conflicts of interest in their included studies, which resulted in a downshift of final score.3 Although we acknowledge this is a difficult criterion to achieve, we feel that this criterion should not be counted as “yes” if not fully met, as it could result in a misleadingly high-quality score.
We conducted a pilot systematic search, which involved a MEDLINE search of all hand surgery systematic reviews from 2003 to 2013 published in Plastic and Reconstructive Surgery. The search identified a total of 90 articles. After screening titles and abstracts, 76 articles were excluded. Eventually, 14 hand surgery systematic reviews were identified. We applied the same quality assessment tool, AMSTAR, on these studies, focusing on the two aforementioned criteria discussed previously. We found that only 21.4 percent (versus 88.1 percent) of studies provided a list of included and excluded studies and none of the studies (versus 81 percent) provided a clear acknowledgment of potential source of support in the systematic review and the included studies.
Researchers and clinicians have the responsibility of providing the highest possible level of evidence, given the significant implications on clinical and policy decision-making. When we assess the literature, the aim is to ensure that the best quality and highest level of evidence is being used. We should avoid overstating any results, as this could affect the progress of the improvement of quality in our surgical specialty by giving a false encouraging image. We find that this unjustified modification of the two AMSTAR criteria has led to inflated results that may have influenced the final conclusion of the study in a misleading manner. Although we feel that these modifications will not help in the improvement of the quality of systematic reviews in the hand surgery literature, the authors may have a reasonable justification for modifying these two AMSTAR criteria.
The authors have no financial interest to declare in relation to the content of this communication. This letter did not require any funding.
Osama A. Samargandi, M.D., M.H.Sc.
School of Population and Public Health
Faculty of Medicine
University of British Columbia
Vancouver, British Columbia, Canada
Haroon Hasan, B.Sc., M.P.H.
Department of Radiation Oncology
British Columbia Cancer Agency
Vancouver, British Columbia, Canada and Pediatric Oncology Group of Ontario
Toronto, Ontario, Canada
1. Momeni A, Lee GK, Talley JR. The quality of systematic reviews in hand surgery: An analysis using AMSTAR. Plast Reconstr Surg. 2013;131:831–837
2. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10
3. Popovich I, Windsor B, Jordan V, Showell M, Shea B, Farquhar CM. Methodological quality of systematic reviews in subfertility: A comparison of two different approaches. PLoS One. 2012;7:e50403
Letters to the Editor, discussing material recently published in the Journal, are welcome. They will have the best chance of acceptance if they are received within 8 weeks of an article’s publication. Letters to the Editor may be published with a response from the authors of the article being discussed. Discussions beyond the initial letter and response will not be published. Letters submitted pertaining to published Discussions of articles will not be printed. Letters to the Editor are not usually peer reviewed, but the Journal may invite replies from the authors of the original publication. All Letters are published at the discretion of the Editor.
Letters submitted should pose a specific question that clarifies a point that either was not made in the article or was unclear, and therefore a response from the corresponding author of the article is requested.
Authors will be listed in the order in which they appear in the submission. Letters should be submitted electronically via PRS’ enkwell, at www.editorialmanager.com/prs/.
We reserve the right to edit Letters to meet requirements of space and format. Any financial interests relevant to the content of the correspondence must be disclosed. Submission of a Letter constitutes permission for the American Society of Plastic Surgeons and its licensees and asignees to publish it in the Journal and in any other form or medium.
The views, opinions, and conclusions expressed in the Letters to the Editor represent the personal opinions of the individual writers and not those of the publisher, the Editorial Board, or the sponsors of the Journal. Any stated views, opinions, and conclusions do not reflect the policy of any of the sponsoring organizations or of the institutions with which the writer is affiliated, and the publisher, the Editorial Board, and the sponsoring organizations assume no responsibility for the content of such correspondence.
The Journal requests that individuals submit no more than five (5) letters to Plastic and Reconstructive Surgery in a calendar year.