Journal Logo

Feature: Evaluating the Evidence

Determining the level of evidence

Nonexperimental research designs

Glasofer, Amy DNP, RN, NE-BC; Townsend, Ann B. DrNP, RN, ANP-C, CNS-C

Author Information
doi: 10.1097/01.CCN.0000612856.94212.9b
  • Free

This series will aide critical care nurses in understanding the basics of research design to determine the strength of a piece of evidence. The previous article in this series explained that not all scholarly evidence is equally effective in determining cause and effect.1 Certain features of the evidence impact the risk of bias and generalizability of its findings, also known as the strength of the evidence.2 Evidence can be organized into hierarchies based on its strength. The prior article in this series reviewed Levels 1 and 2 of the Johns Hopkins hierarchy of evidence, which included experimental and quasi-experimental research.3 This installment covers nonexperimental research appraisal (Level 3). (See Evidence hierarchy.)

In nonexperimental research, there is no manipulation of an independent variable, no requirement for a control group, and no random group assignments as in experimental and quasi-experimental designs.3,4 Another term for this type of research is observational because the researcher observes natural occurrences without intervention. Many nonexperimental research designs exist. Each design is typically classified by purpose (to describe, predict, or explain) and time (prospective—applying to the future, retrospective—applying to the past, or longitudinal—over time). The research design choice is driven by the research question's intent to describe, predict, or explain the variable.3,4

This article will provide critical care nurses with guidance to identify key features of the most-common nonexperimental research designs. This installment will again use as an example the question: What is the effect of caffeine on nursing medication errors? Because this article does not address experimental research, cause and effect relationships between the variables cannot be determined. The research question wording must change to align with the specific nonexperimental design.

Descriptive research

Descriptive research generally aims to describe characteristics, behaviors, and conditions of individuals and groups.4 Descriptive research can be retrospective, prospective, or longitudinal. A typical descriptive study looks at a single sample.5 Surveys are frequently used in descriptive research to provide an overall picture of a group's characteristics. Surveys require that the researchers identify variables of interest and determine how variables are measured.5 For example, perhaps researchers simply want to describe how much caffeine nurses consume. It would be difficult to do this retrospectively, unless nurses had been documenting their caffeine consumption. However, researchers could use a cross-sectional study that would involve studying a sample of nurses at a single point in time and asking them to generalize about their caffeine consumption. Researchers could also conduct a longitudinal study by asking nurses to journal their caffeine consumption over time. Alternatively, a prospective observation of caffeine consumption behaviors in nurses would involve making observations of nurse caffeine intake in real time. Each of these designs would represent a typical descriptive study because the findings of this study would help researchers describe the caffeine consumption in the sample group of nurses.

Alternatively, researchers could conduct a comparative descriptive study. As the title indicates, a comparative descriptive study allows researchers to compare groups and look for naturally occurring differences.5 In this case, researchers might conduct a survey asking nurses both about their caffeine consumption and if they have ever made a medication error. The naturally occurring samples could be nurses who have made an error and those who have not. Researchers could compare caffeine consumption between the two groups. Again, it is important to note that the only purpose in doing this would be to describe caffeine consumption and make comparisons between the groups. This research design does not allow for any conclusions about a causal relationship between caffeine and medication errors to be drawn. The results of this descriptive study may justify a future experimental study, such as a randomized-controlled trial to explore the cause and effect of caffeine on medication errors.

Qualitative studies are descriptive studies used to understand concepts that have not been completely described.3 Instead of the number-based data collection used in quantitative designs, qualitative designs use narratives to better understand the natural experiences of the participants. For example, researchers could interview individuals or hold small focus groups on the experiences of nurses who have made medication errors to look for similar themes among participants. The presence or absence of caffeine or fatigue might be a common theme that emerges. The research data collected for qualitative studies use interviews, focus groups, or individual stories that are analyzed for themes. Qualitative research designs include phenomenology, ethnography, historical research, grounded theory, and case studies.3 The preferred approach is determined by the research question.6 The texts referenced in this section provide in-depth understanding of qualitative research designs.3-7 Like the meta-analysis of quantitative research, a meta-synthesis is a review and analysis of multiple qualitative studies.3

Figure
Figure:
Evidence hierarchy

Exploratory research

Exploratory research is a type of observational research that aims to investigate the relationships among two or more variables.4 When researchers have a hypothesis they want to test, they can do so through an exploratory design. Exploratory studies can also be retrospective, prospective, or longitudinal. Exploratory research designs include correlational, predictive correlational, case-control, and cohort studies.

Correlational studies seek to measure the degree of association among variables.4 That is, do they increase or decrease together, in opposite directions, or have no association? For example, researchers may want to look at the relationship between the amount of caffeine a nurse consumed and the number of medication errors they have made. This could be done retrospectively if there were data on caffeine consumption, but more likely this would require a prospective or longitudinal design. Ultimately, researchers will determine if there is a relationship between caffeine and medication errors and describe how strong that relationship is. Perhaps it is determined that as caffeine consumption increases, medication errors decrease. This would be an example of an inverse relationship. Although this relationship is interesting, it would be incorrect to say that increasing caffeine consumption will decrease a nurse's risk of making an error. It could be that some other variable such as the nurse's years of experience better explain the relationship. The reason for the relationship between caffeine and medication errors may exist because nurses with more experience consume more caffeine; the lack of medication errors may not be a direct reflection of the effect of caffeine.

Predictive correlation studies are similar, except they take multiple independent variables into consideration to develop a model for understanding when a certain outcome is likely.4 For example, researchers may desire to predict when a medication error (dependent variable) will occur. There may be reason to believe (a theory) that not only is caffeine involved, but also nurse experience, nurse education level, patient acuity, and patient ratios. A statistical technique called regression will calculate the likelihood that a nurse will make an error given these independent variables, and which of these variables make the greatest contribution to the risk of error.4

Case-control research is a method in which a case, or individual with the condition, is identified and compared with a control, or individual without the condition. This is a retrospective design because the investigator looks back in time to gather data about what may have put the case at risk for developing the condition.4 Typically, this is a useful design for conditions that rarely occur, as it is easier to start with identified cases and work backward than it is to wait and see who will develop that condition. It is unlikely that a researcher would use a case-control design when exploring medication errors, but it is possible. In this example, the researcher would identify nurses who have made errors (cases) and those who have not (controls). Subsequently, he or she would gather information from documents, interviews, and questionnaires to identify ways in which the groups differ that may have put them at risk for medication errors.

Cohort studies follow a group of individuals over time who do not start with the outcome of interest and observe to see who will develop the outcome or disorder. Often, cohort studies are specifically interested in how certain exposures impact the risk of developing a certain outcome. The example of caffeine and medication errors would not be a good fit for a cohort study, but the research question could be changed to look for a relationship between bullying and medication errors. In this case, researchers could identify a cohort of new graduate nurses who have not yet experienced bullying. They would collect data over time on medication errors and experiences of being bullied to determine if those who had been exposed to bullying were more or less likely to make medication errors. Although cohort studies still cannot show cause and effect, they can establish a time sequence, and provide the relative risk of developing a condition based on exposure, which provides greater evidence for causality than other observational designs.

Nonexperimental research limitations

Although nonexperimental research studies provide valuable information on the variables under study, they are ranked lower (Level 3) on the hierarchy of evidence because the researchers may be unable to control for competing explanations for their findings. This can introduce error or bias when conducting observational studies. Nonexperimental studies lack the design strength gained from manipulation, randomization, and control groups found in experimental designs (Level 1). There may be differences in the nonrandomized groups under study that can affect the study outcomes, as seen in the caffeine example. Also, because this example contains preexisting groups that are not randomized, one group may not represent other groups of nurses who make medication errors, so the findings cannot be generalized beyond the studied group.

Conclusion

The previous series installment compared Level 1 research with DNA evidence, and Level 2 research with fingerprint evidence. Following this analogy, Level 3 observational studies are like eyewitness testimony. The eyewitness accounts may vary because of memory lapses or unintentional recall errors, or there may be multiple eyewitnesses with different versions of the same event. Thus, a single eyewitness cannot make the case. Similarly, a single piece of Level 3 evidence alone is insufficient to draw a reliable conclusion.

Because nonexperimental research is often more feasible, and sometimes more ethical, to conduct than experimental research, it is likely the most readily available form of research to answer clinical questions. However, nurses must be aware of the pitfalls of relying on nonexperimental research. When added to stronger evidence, or in addition to multiple other sources of Level 3 evidence drawing the same conclusion, nonexperimental studies provide important information for evidence-based decision-making. The next article in this series will cover nonresearch evidence that nurses can also include as information sources.

REFERENCES

1. Glasofer A, Townsend AB. Determining the level of evidence: experimental research appraisal. Nurs Crit Care. 2019;14(6):22–25.
2. Melnyk BM, Fineout-Overholt E. Evidence-based Practice in Nursing & Healthcare: A Guide to Best Practice. 4th ed. Philadelphia, PA: Wolters Kluwer Health; 2019.
3. Dearholt SL, Dang D. Johns Hopkins Nursing Evidence-based Practice: Models and Guidelines. 2nd ed. Indianapolis, IN: Sigma Theta Tau International; 2012.
4. Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. 3rd ed. Upper Saddle River, NJ: Pearson Prentice Hall; 2009.
5. Burns N, Grove SK. The Practice of Nursing Research: Appraisal, Synthesis, and Generation of Evidence. 6th ed. Philadelphia, PA: Saunders Elsevier; 2009.
6. Cresswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. 3rd ed. Thousand Oaks, CA: Sage; 2013.
7. Sawatsky AP, Ratelle JT, Beckman TJ. Qualitative research methods in medical education. Anesthesiology. 2019;131(1):14–22.
Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved