Secondary Logo

Journal Logo

FEATURES

UNTANGLING THE TANGLED

A Practitioner’s Guide to Evaluating and Understanding Research

Kilpatrick, Marcus Ph.D.; Green, Matt Ph.D.

Author Information
ACSM’s Health & Fitness Journal: March/April 2012 - Volume 16 - Issue 2 - p 13-18
doi: 10.1249/01.FIT.0000413043.30989.ea
  • Free

Figure
Figure
Figure
Figure

Research in exercise science has grown exponentially in the last few decades, with outcomes clearly indicating that physical activity and other lifestyle-related factors can significantly transform health and well-being. This research does make its way into the hands of practitioners, but, too often, practitioners are not provided with this crucial information in a format that is relevant and easily understood. Research reported in scientific journals often includes terminology and statistical jargon not easily understood by many health and fitness professionals. Although a select few journals, such as ACSM’s Health & Fitness Journal®, focus on articles that are intended for practitioners “in the trenches,” it is important for professionals to be equipped with knowledge and tools that allow for better understanding and application of concepts from published research in exercise science. The current article endeavors to serve as a primer of sorts for the professional who wants to be in a better position to decipher research and leverage its results in everyday practice.

RESEARCH CATEGORIES

An excellent starting point in any discussion of research is the recognition that the research process takes many forms. Although varied in the approaches, all forms of research share the common goal of developing knowledge by way of systematic investigation of some phenomenon of interest. The two major overarching methods of research are quantitative and qualitative. Quantitative research is based on the assumption that important questions are best answered using mathematical principles and statistical techniques. In contrast, qualitative research approaches a research question in a more personal and narrative manner. Qualitative approaches to research include case studies and historical research, in addition to an array of approaches in which unique personalized data are obtained. Of the many varieties of qualitative research, only case studies are used considerably in research related to exercise and health. Case studies use a detailed descriptive analysis that typically focuses on the clinical aspects of some phenomenon. For example, a case study might profile the assessment and intervention of some injury or illness in a way that provides a model for addressing similar cases in the future. Although qualitative approaches to research are becoming more accepted and commonplace, most exercise research uses traditional experimental design, survey or questionnaire research, and epidemiology. Likewise, most published research related to exercise and health appears in journals that primarily publish quantitative research findings. As such, this article focuses on helping practitioners understand quantitative research designs and results.

Figure
Figure

CONSIDERATIONS OF PUBLICATION TYPE

One initial consideration for practitioners interested in becoming better consumers of health and fitness research is the nature of the publication itself. The health and fitness field is evolving toward becoming a discipline that is based around evidence-based practice, which is a positive direction for the profession as it moves forward in gaining credibility with other health professionals. These changes to our field underscore the need for health and fitness professionals to base decisions and actions on evidence provided by science that is published in research journals. As such, practitioners should seek to obtain professional practice information from high-quality peer-reviewed publications rather than trade or professional magazines. One rule of thumb for the practitioner is that journals associated with professional organizations and library databases are generally better sources than professional magazines and trade journals available through retail outlets. It should be noted, however, that the quality of research journals can vary significantly, and some measure of skepticism is appropriate. Regardless, practitioners can generally feel confident in the information provided by scientific journals that require review of the work by experts in the field before publication. Listing all of the good research journals in the health and fitness discipline is not practical for this article, but a few good sources include Medicine & Science in Sports & Exercise®, Journal of Strength and Conditioning Research, Research Quarterly for Exercise and Sport, Journal of Sport Sciences, and the International Journal of Sports Physiology and Performance.

TYPES OF RESEARCH ARTICLES

Research writing can take many forms, but most articles can be placed into one of a few categories. The most common type of published research is the data-based research article, which provides the details of a single research study and typically includes the following sections: Abstract, Introduction, Methods, Results, and Discussion. Guidelines designed to assist with comprehending each section is provided in a later section. Another major type of research article is the review article, which is a scientific report that organizes and summarizes previously completed research studies. These articles sort through existing research and describe the current knowledge base in the field as determined by an expert in the field of study covered in the article. This kind of article is excellent for learning what experts in the field believe is the current state of knowledge on a particular topic. The last major category is the meta-analysis, which is an extension of the review article and attempts to analyze all existing research from a mathematical perspective in hopes that bias can be limited, which allows the data to speak for itself. Each type of scientific article is valuable. This article endeavors to help practitioners break through some of the challenges associated with data-based research articles and develop a better understanding of how to read these articles in a way that will benefit professional practice.

ANATOMY OF A DATA-BASED RESEARCH ARTICLE

A primary aim of this article is to provide insights into how to best read and interpret research articles, and a good first step is a quick review of how articles are organized. The typical research article is written so that the story of a research study unfolds sequentially. Most research articles have four narrative sections in addition to the abstract, tables, and figures. Better comprehension of research findings is facilitated by an increased familiarity with each section, which allows the reader to contemplate important questions while reading the article. Table 1 provides a list of questions that should be considered when practitioners read research articles.

TABLE 1
TABLE 1:
Key Questions to Ask When Reading Research Articles

The first part of most research articles is the Abstract, which serves as the abbreviated version of the research project. Abstracts also are the part of the article that is typically identified when searching databases such as PubMed, SportDiscus, and Google Scholar. Providing a brief overview of the global purpose, methods, and take-home message or general conclusions from the project, the Abstract allows the reader to determine whether he or she would like to read the entire article. The first narrative section is the Introduction. This section provides background information, with the goal of briefly articulating what is both known and unknown about a topic area. The Introduction section is written in a way that tells the story of existing research, often including descriptions of previous studies that sometimes have found contradicting results. The Introduction section generally is concluded with a clear statement of the new study’s importance and purpose.

The Methods section describes the details of the research project in terms of who, what, where, and when. A description of variables such as age, gender, body composition, and physical activity levels of the research participants helps the readers form an opinion of how relevant the research outcomes might be within their professional work. This section also is the first place where the reader may encounter a table of basic data related to the research participants. The Methods section also describes the step-by-step procedures used to complete the project and indicates to the reader whether the study involved a single point in time (i.e., a cross-sectional or acute design) or instead considered changes during a significant period (i.e., longitudinal design). Close inspection of the methods and procedures will help the practitioner determine how relevant any findings might be to his or her work.

The Results section describes the outcome of the study and typically includes the statistical findings. A narrative description of the results is important in many respects but is likely of lesser importance to the practitioner in comparison with the interpretation of related tables and graphs. Being able to read and understand these figures is crucial for the practitioner interested in being a better consumer of research. A picture may often be worth a thousand words, but even more important, it is frequently more easily interpreted.

The final section in most research articles is the Discussion, which functions to describe how the research findings add to the existing base of knowledge. A comparison of the current project with previous research is typically the first part of the discussion. This type of information is intended to help the reader see how conclusions from the old and new research results may agree or disagree. The Discussion section also typically describes the relative strengths and weaknesses of the project. Research worth publishing provides new insights, and careful attention should be paid by the reader to see what might be “new and great” in any published article. Most often, these strengths relate to changes in research methods or the sample of participants used. Relatively small changes to the design of the study can produce information that can be quite beneficial. An example would be a study that examines females as participants when all previous research on the topic had exclusively used males. Likewise, good research readily points out the areas where the study is limited in terms of research design features. The Discussion section also makes recommendations for future research to provide clarification on issues not yet settled. A final component in the Discussion sections of many articles covers the practical applications of the research findings. Inclusion of this kind of information describes for practitioners how this new information can be used to help real people in real situations. This short section can be a rich source for the practitioner in that it is tightly focused on the most relevant and take-home information from the study.

EVALUATING THE RESEARCH DESIGN

One prominent feature of a good research design is the inclusion of one or more control groups that provide researchers with increased confidence in their findings. A simple and important feature is the use of a control group alongside one or more experimental groups. Suppose, for example, a researcher wants to compare strength gains in response to a 6-month resistance exercise program in a group of adolescent boys. High-quality research generally would compare the changes observed in the group receiving the weight training intervention with a group not receiving the intervention. This step could be important because strength gains can occur in the absence of exercise training simply because of normal growth and development, particularly in this age group. The inclusion of a control group helps separate the impact of the intervention from responses that may be caused by the passage of time or some other variable.

A related issue is the inclusion of placebos in research design. A placebo often is referred to as a “dummy treatment,” whereby one group is given a legitimate intervention or treatment and a comparison group is given a placebo of some kind that by all appearances is the same as the experimental intervention. This type of design is used to determine whether differences observed are caused by the experimental treatment or simply the power of suggestion and is important because belief in the intervention itself can produce positive results. Placebos typically are used in conjunction with blinding procedures. In a blinding procedure, research participants do not know whether they are receiving the legitimate treatment or the placebo. Such blinding procedures allow the researcher to separate the impact of the anticipation of the intervention from the actual intervention. Sports nutrition research provides ready examples for these design elements. Typical blinding procedures involve informing all participants in an exercise study that they would receive a supplement, such as caffeine, when in actuality only one half of the participants received caffeine and the other half received a placebo with no caffeine. This procedure controls for the possibility that belief in the supplement alone might increase exercise performance independent from the actual physiological effects of the supplement. One additional level of control that is often included in such designs is the use of double blinding procedures. Double blinding occurs when the assignment to the placebo or intervention group is hidden from both the research participant and select members of the research team involved in direct implementation of the intervention. This added step serves to limit the negative impact of researcher bias on study results. Each of these design elements intends to identify the impact of a particular intervention after controlling for other factors that may otherwise limit the ability to appropriately interpret research findings. In a more general sense, these features improve the quality of the research project and their presence should increase the reader’s confidence in the study results.

INTERPRETING THE SIGNIFICANCE OF RESEARCH RESULTS

Among the most important variables to understand when reading research is the concept of significance. Significance can exist both from a statistical and practical perspective. Statistical significance is present when it is determined that some observed relationship or difference for two variables is likely to be present for a reason other than chance. Statistical significance is based on the mathematical principle of probability. This type of significance most typically is reported in research articles using P values. The P value represents the likelihood that an observed relationship (when correlating variables) or difference (when comparing variables) is caused by some factor other than chance. Generally, a value of P < 0.05 is considered necessary to conclude that a finding is statistically significant, but researchers do sometimes select a more liberal or conservative standard. The assumption made when the value of P > 0.05 or other criterion is that any difference observed was caused by chance rather than reliable differences between groups. As an example, a value of P < 0.05 indicates to the reader that there is a less than 5% chance that the observed correlation or difference is caused by chance. Therefore, the reader can be more than 95% confident that the observed finding is in fact legitimate. Such levels of confidence typically are adequate to inform decisions related to professional practice.

Figure
Figure

The second type of significance is referred to as practical significance. Although statistical significance describes the likelihood that the observation is “real,” practical significance exists when the observation is large enough to be of value in everyday practice. Practical significance typically is expressed as an effect size, which reflects the size or magnitude of the observed difference rather than the statistical likelihood of any difference. In this regard, practical significance can be thought of as the common sense difference observed in data. Sometimes, the effect size is discussed explicitly in a research article, but this is not always the case, and practitioners will be better positioned to read research effectively with this view of significance in mind.

Effect size can be calculated by subtracting one mean value from the comparison mean value and dividing that number by the average of the two SD values. Effect size values that are close to zero generally are considered to reflect little practical significance, whereas values equal to or greater than plus or minus 0.5 are believed to be of greater practical significance. It is common to consider effect size as a measure of practical significance when statistical analysis reveals significant differences.

Mock research data related to significance are provided in Table 2 and the Figure. Both visuals relate to a fictitious research study that examines the impact of caffeine on muscular exercise performance. Table 2 provides descriptive data related to the research participants, and the Figure provides the primary results of the study. A preliminary assessment of the information provided within Table 2 suggests that this research sample is relatively young, mostly male, mostly healthy in weight, and relatively fit. A more detailed inspection of Table 2 with respect to gender is provided in the far right column with P < 0.05, indicating a significant difference between male and female groups and P > 0.05, indicating no significant difference between the male and female groups.

TABLE 2
TABLE 2:
Interpreting Descriptive Statistics*

For these data, the important points are that male and female groups were significantly different in aerobic fitness, but not different on age, body mass index (BMI), or upper body strength. The lack of a significant difference for upper body strength is somewhat surprising and suggests that the females in the study are highly experienced with resistance exercise in comparison with their male counterparts. Such a finding should be noted and considered when making summary assessments of how the findings of the study should be interpreted. The Figure shows the means and SD values for repetitions per trial in response to three different levels of caffeine ingestion. The asterisk indicates that the statistical analysis revealed a significant difference for repetitions completed, and that the high dose resulted in more repetitions than the low and moderate doses. A global view of these data leads the reader to conclude that a high dose of caffeine has a significant influence on repetitions when compared with low and moderate doses, which are not different from each other.

Figure
Figure:
Interpreting graphs/figures. Repetitions to failure among trials where participants ingested various doses of caffeine: low dose (2.0 mg/kg body weight); moderate dose (4.0 mg/kg body weight); high dose (6.0 mg/kg body weight). The figure depicts means and SE values. The asterisk (*) indicates that the high-dose trial is significantly different from the low- and moderate-dose trials, which are not different from each other (P < 0.05).Figure. Notes. Orient yourself and focus on what is represented by each axis and the values. In this case, the vertical or y axis describes repetitions to failure and the horizontal or x axis details results for three different conditions.Figure. The columns represent the mean values for the group for each trial as labeled. The small bars at the top of each column represent the SD or the SE associated with the mean. SE values represent variation around the mean that is based on the sample size and the SD.Figure. Asterisks or other symbols reflect significant differences. Authors of research articles represent these differences in a variety of ways. Practitioners are encouraged to carefully read the notes associated with each figure and table so that important details related to the visual can be interpreted appropriately.Figure. The take-home message from viewing this graph is that a high dose of caffeine provides a benefit over both a low dose and a moderate dose in regard to muscular endurance tasks, while low and moderate doses are not significantly different from each other.

A final note from the Figure helps describe both statistical and practical significance. The data indicate that a high dose of caffeine facilitates better performance than a low or moderate dose, but a closer look at the comparison between the low and moderate doses provides a good example of significance. Although the results described in the footnote for the Figure indicate that the difference between the low and moderate dose was not statistically significant (P > 0.05), suppose for a moment that the comparison of the low-dose trial (mean = 12 repetitions; SD = 4) and the moderate-dose trial (mean = 13 repetitions; SD = 6) produced a corresponding value of P = 0.04 (instead of the actual value that was above 0.05). Such a result would indicate a statistically significant difference (meeting the required value of P < 0.05). However, it could be argued that a 1-repetition difference is not practically significant, and a quick calculation of the effect size supports this possibility. More specifically, the calculated effect size of 0.2 (mean difference of 1 divided by the average SD value of 5) would be described as a small effect and not particularly meaningful in terms of professional practice. Generally, small effect sizes are the result of small mean differences and/or relatively large SD values. The primary message is that issues related to statistical and practical significance should be thoughtfully considered when reading research studies so as to maximize the value of the findings to professional practice.

SUMMARY

The health and fitness field is primed for continued growth and evolution. Growth is anticipated because of the expanding public health burden created by poor health habits. Evolution is already underway as professional organizations move toward raising the standards for entry into the field and the possibility of widespread licensure moving forward. These are exciting times for the health and fitness field, and the dedicated professionals working on the front lines are taking on new and important responsibilities within a variety of organizations and communities. Increased expectations come with these new opportunities, and one of these expectations is that health and fitness professionals will be knowledgeable and current with respect to the research in their disciplines. The nature of science is that cutting-edge research is published in data-based articles several years before more widespread dissemination of this information by way of policy statements and position stands. This sequence of events requires professionals in our discipline to be aware of the latest research findings so that this information can be used in a timely manner to benefit the individuals they serve. This article has endeavored to provide a stronger foundation for all the dedicated professionals who desire to be better positioned along the front lines in the ongoing battle against sedentary living and unhealthy lifestyles.

CONDENSED VERSION AND BOTTOM LINE

Scientific journals provide findings from cutting-edge research that too often do not make their way into the hands of the practitioners who need this information to be more effective professionals. Reading and making a meaningful interpretation of research in our discipline are challenging. The contents of this article are intended to both encourage and empower health and fitness professionals to take on this challenge.

Recommended Readings

Dictionary of Statistics and Methodology: this resource is a nontechnical dictionary of commonly used research and statistical terms; Paul Vogt; Sage Publications.
    SportScience: this resource is a Web site (sportsci.org) that provides descriptions and definitions of numerous research designs and statistical concepts.
      PubMed: this resource is a free database Web site (pubmed.org) set up by the U.S. National Library of Medicine that allows individuals to read journal abstracts and some full-text articles of published research.
        Keywords:

        Research; Statistics; Significance; Journal Articles

        © 2012 American College of Sports Medicine.