Simulation in Healthcare recently published summaries and monographs from two important conferences1,2 that each aimed, in different ways, to lay out the current state of research on simulation in healthcare and to consider the kinds of research needed in the future. In commenting on these groundbreaking events in the August 2011 issue of Simulation in Healthcare, I wrote:
In the future it is likely that we will need to mobilize yet larger resources to provide more definitive answers to the big questions about simulation. As we do so we will need to articulate more fully the key themes and questions and to demonstrate the linkage between them and our projects. … how we might organize to bring the visions of the Summit to fruition and to convince policy makers of the need to fund the long-term research agenda.3
This article explores some concepts for taking bold new steps in strategic thinking about simulation research. Some steps we can begin now, but others may take several years to begin in earnest, with programs of potential funding and research that might not start until the second half of this decade or later. As this is written (September 2011), the political climate in Washington and other capitals is confrontational and uncertain. Yet, we must take the long view and believe that rational thinking about the key themes and questions about simulation in healthcare will find a constructive policy and funding climate somewhere, now or in the future.
Historically, research concerning simulation in healthcare has depended largely on innovations and ideas of individual investigators. In the jargon of research strategy, this is known as an “investigator-initiated” approach. This strategy has propelled simulation research into many diverse avenues of modalities of simulation and its applications with projects touching many of the “cells” of the 11-dimensional space of simulation.4 The good side of this diversity is the breadth of exploration of important topics, target populations, and purposes of simulation. The bad side is that it is difficult to put this diversity into a coherent scientific and policy context. Any particular project addresses only a small part of a much larger whole. One of the challenges for our field, as we seek to engage in more robust and comprehensive research, is to develop comprehensive models of our field through detailed strategic analysis of the research landscape. A structured standard strategic framework would help investigators, young and established, to select and to frame their individual research interests and proposals within the larger context. Perhaps most importantly, it would allow us to better explain—to funders and the public—how individual pieces of work relate to the big picture of understanding how and why simulation can improve the safety and quality of healthcare for the people of the world.
Simulation and patient safety have borrowed many good ideas from nonmedical scientific and technical fields. In considering the operational paradigm for safety in intrinsically hazardous work, we looked to aviation, nuclear power production, and the military and adapted concepts of organizational safety, human factors, and indeed of simulation itself. The deliberations at the recent research strategic planning sessions have indicated that there are many unsolved questions about simulation in healthcare, the answers to which may well require us to propose and prioritize larger, more expensive, and lengthy projects along with a plethora of small ones. As I thought about how to do this, I found myself resonating with documents and processes from yet another technical paradigm, that of space sciences, a field that I happen to follow closely. There are of course many approaches to strategic planning of research, and the methods I will describe are not unique to astrophysics and planetary science, although that is how I found out about them. In this article, I will describe what makes two particular processes attractive to me.
SCIENCE TRACEABILITY STRUCTURES
In astrophysics and planetary science, key data gathering programs—often space-based “missions”—are so expensive that it is critical to justify explicitly how the scientific instruments carried will in fact address scientific themes and questions deemed to be of fundamental importance by the scientific community. To document clearly how their proposals relate to the structure of these key scientific questions, scientists and mission planners use a “Science Traceability Structure”5 (STS) that explicitly links broad themes to relevant scientific goals (collectively these two levels constitute “key questions”), on to specific objectives, and thence to particular instruments on particular missions or experiments. The STS per se does not define the priority of the key questions—that prioritization takes place in other forums including the Decadal Surveys described later in this article. The traceability concept seems to have percolated into science planning from “requirements traceability” in systems engineering and software development.6 I have not seen any similar method for explicitly tracing scientific goals to projects and methods used in the healthcare arena.
Discussion of science traceability is prevalent in many proposals for and presentations about space missions. I found a particularly clear explanation and application of a STS in a presentation of the scientific justification for a proposed mission to Jupiter to address “the search for habitable moons of giant planets” by sending an orbiter to Europa, a moon of Jupiter that is likely to have a deep subsurface ocean that might even harbor life.7 This theme is a deep scientific question, in principle having a definitive set of answers. There either is or isn’t an ocean under Europa’s crust, and if there is, it either does or doesn’t have characteristics that could support life as we know it. Empirically verifying the facts will be very difficult but there is no fundamental reason why they cannot be determined. It may be true that some of the questions we have about simulation in healthcare are more ambiguous than these, and perhaps we sometimes have more practical objectives that do not necessarily require a definitive answer. Hence, some of the simulation research endeavor may not be “science” in the way that STS envision. But even if we think of them only as “concept traceability structures” the techniques of formally linking themes, goals, and objective to the programs, projects, and methods to address them are very applicable to research on or using simulation in healthcare. Hence, I have chosen to stick with the original nomenclature of STS.
Table 1 and Figure 1 illustrate the concept of a STS. Table 1 shows the STS levels both for the Europa mission and as adapted to simulation. Table 2 gives a “straw man” example of elements of different levels of an STS as it might be created for articulating the linkage between themes, goals, and objectives to programs, projects, and methods for simulation in healthcare. Table 2 presents four example grand themes about simulation and then drills down from one of them to formulate an example “thread” of research reaching all the way down to a handful of possible methods to address a very specific subquestion. In this example, each level of the STS has about three to five elements. In a real STS, there might be even more elements at certain levels. Nonetheless, assuming about four elements per level, even a simple example STS would articulate over 250 programs of research (44) and just over 1000 projects (45), each of which might make use of several different methods. When all threads of a particular program are put together, it forms a Science traceability matrix, which is a common format for representing the coverage of the various methods to address the relevant objectives, goals, and themes.5 The straw man STS is only notional to illustrate how an STS could be applied to simulation in healthcare. Defining the ideal set of themes, goals, and so on for simulation in healthcare, complete with a fair representation of elements at every level, would be a serious consensus-forming endeavor requiring comprehensive engagement of the simulation and stakeholder communities.
There is no single STS covering all of human knowledge and inquiry; an STS for one field likely overlaps with other fields. Indeed, some of the general themes and goals about understanding simulation might be common to a wider variety of arenas of human performance and not confined only to healthcare. Nonetheless, there is still value in articulating a coherent structure for a specific portion of inquiry and application such as healthcare even if parts of it are replicated in other fields.
Once created, a simulation STS could be used to categorize research in either a top-down or a bottom-up fashion. Given the diverse and often decentralized systems of funding simulation research, perhaps the bottom-up approach is the easiest to imagine. Once the community has articulated the structure of key themes, goals, and objectives, individuals wishing to propose investigator-initiated programs and projects, or those who develop new methods, could explicitly demonstrate how the results of their work would illuminate portions of the STS’ key questions. Currently, investigators do this in the introduction sections of manuscripts or in the background section of grant proposals, but these attempts are often ad hoc and incomplete. The healthcare simulation community can find value in using a more formal structure to describe the interrelationships in our research.
The top-down way to use a STS would be for funding agencies to explicitly formulate requests for proposals that concentrate on specific goals and objectives within given themes. Funders like the Agency for Healthcare Research and Quality (AHRQ) in the United States currently provide background rationales in their requests for proposals, but these do not link to any comprehensive structure, rarely show the interrelationships between projects, and do not have the force of consensus behind them. Perhaps of greatest importance, the use of a STS would not only help our science but also be a very important vehicle for the simulation community—as it has been for the astrophysics and planetary science communities—to convince policy makers to invest in answering important questions.
DEFINING THE HIGH LEVEL THEMES AND GOALS
How should the highest level themes and goals be delineated? How can we make the best case for larger, longer, more robust studies? In astrophysics, planetary science, and earth science, such big issues are defined by formal consensus-forming processes within the scientific community. In the United States, one of the main processes for this is called a Decadal Survey.8 Note that the term “Survey” in this context means a “study” or rigorous survey of the field. I found the most recent ones for astronomy and astrophysics9 and for planetary science10 interesting reading. A Decadal Survey is requested and funded about every 10 years by the relevant federal funding agency (eg, NASA), but it is conducted under the neutral auspices of the National Research Council (NRC—made up of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine). A Decadal Survey encompasses the following:
- i. An extensive state-of-the-science summary (considerably more comprehensive than those undertaken in the recent simulation in healthcare research strategy events).
- ii. An articulation of a strategic scientific program (with at least the big picture elements of a STS).
- iii. For big-science areas with major equipment or facilities, a mission-by-mission analysis and recommendation for which large programs should/should not be funded or with what priority programs and missions should receive support. Typically research programs in these fields are divided into several categories:
- Missions (where costs are spread over many years)
- Flagship level (>$900 million; about 1 per decade)
- New Frontiers level ($450–$900 million; about 2 per decade)
- Discovery level (<$450 million; about 5 per decade)
- Research and Analysis
- Nonmission science measurements (eg, using ground-based instruments)
- Programs for young researchers
- Generic technology development for likely future missions
Funding for simulation in healthcare could be one or two orders of magnitude lower than for planetary science (eg, a flagship program might be $10–100 million) and still be one to two orders of magnitude greater than current funding levels. An important feature of simulation research is that, unlike pure astronomical science, it also offers at least the hope of facilitating long-term cost savings to the federal government.
The key features of the Decadal Surveys that I believe should be adopted for a similar study for healthcare simulation research are the following:
- i. They are repeated about every 10 years rather than being one-shot efforts.
- ii. They are run by the scientific community not by the funding agencies or regulatory body (unlike, say, the National Institutes of Health Roadmap process).
- iii. They have dedicated professional staff to assist the scientists in the Survey’s work.
- iv. They undertake intensive efforts to obtain ideas and guidance from the broad scientific community.
- - Many of the meetings of the Survey Steering Committee and Panels are open to the public; meeting minutes, transcripts, or videos are available on the web (see http://sites.nationalacademies.org/SSB/CurrentProjects/ssb_052412). However, the final deliberations of the Steering Committee are private.
- - The Survey Steering Committee and the various Topic Panels invite oral presentations from the scientific community and they hold public forums and town hall meetings at regional locations nationwide and at major national and international scientific conferences.
- - They openly solicit the submission of “white papers” from individuals and organizations in the community (there were 199 papers filed for the Planetary Science Decadal Survey—see http://www8.nationalacademies.org/ssbsurvey/publicview.aspx).
- v. They tackle hard issues of prioritization and where necessary conduct independent cost estimation for very expensive missions or programs.
- vi. They strive for a balance between large programs (to answer intrinsically complex questions) and small- and medium-sized programs.10,11 In fact, the recent Decadal Surveys insisted on this balance, even at the risk of “descoping,” delaying, or cancelling some otherwise highly rated medium and large missions.
- vii. They explicitly aim to develop new scientific talent both in the Survey process itself and in the funding of the various programs considered.
- viii. They use independent external review of the Survey reports before final revision and publication.
- ix. Their results are widely accepted, are referred to ubiquitously in scientific presentations and publications, and are highly influential to policy makers and legislators who seek guidance about where best to invest research dollars.8
The Decadal Surveys are major undertakings, typically taking 2 years of sustained effort to complete, a magnitude considerably greater than that mobilized by the simulation in healthcare community for the two conferences in the last 2 years. For example, the Planetary Science Decadal Survey was made up of a 70-person Committee—a 16-member Steering Group and 54 additional experts, the latter group organized into five Topic Panels (in that case inner planets, Mars, giant planets, satellites of giant planets, and primitive bodies). After the literature review and the various community inputs are obtained, assessments and recommendations are then integrated by the Steering Group. The first full draft of the report is sent for independent confidential external review in accordance with procedures approved by the NRC’s Report Review Committee, and the report is revised accordingly before final release. Decadal Surveys are published by the National Academy Press as “books” but they are downloadable as files for free. Decadal Survey endorsement of particular themes, goals, and objectives, or a Survey’s declaration of a high priority for a mission, does not guarantee funding—especially in times of lean budgets and political confrontation—but the recommendations often mobilize support for missions in the legislative branch and the Surveys typically guide expenditures by funding agencies within general programs.11
The resources required to conduct a Survey are nontrivial but are probably less than the cost of a single AHRQ simulation grant (about 1 million US dollars). The commissioning agency foots the bill for the NRC to provide infrastructure and professional staff for the effort and to support the travel costs of Committee members and staff. The scientific community provides the membership of the Committee, which performs much of the intellectual work; members usually serve without compensation. The most expensive part of the most recent astrophysics Decadal Survey was contracting for independent cost analyses of large missions, a process that is not likely to be necessary for simulation programs (Roger Blandford, PhD, personal communication, 2011). The Decadal Survey is a US process, but in European space sciences there are similar strategic studies although not necessarily using exactly the mechanisms of the Decadal Survey (see for example the European Space Agency’s Cosmic Vision: Space Science for Europe 2015–202512).
I believe that a study using the methods of a Decadal Survey would be a logical follow-up to the results of the Society for Simulation in Healthcare (SSH) Research Summit and the SESAM/SSH Utstein-style meeting. A Decadal Survey is more comprehensive, is conducted over a longer period of time, provides multiple mechanisms to engage and receive input from the scientific community, and undergoes more complete peer review of the end product. Planning for research endeavors on a decadal basis—including studies that might even take 1 or 2 decades to organize and execute—and doing so decade after decade is a sign of a serious and mature field.
To answer high-level questions about simulation may require “programs” of much longer duration, larger size and scope, and complexity than our community has ever fielded; essentially the healthcare equivalent of planetary science flagship programs (albeit at much lower cost). I propose that the next major step for strategic planning of simulation research in the United States should be to persuade AHRQ to commission and fund the Institute of Medicine (the relevant component of the NRC) to conduct the equivalent of a Decadal Survey for simulation in healthcare, using the methods that have made such Surveys so valuable in other fields. Organizations like SSH, Advanced Initiatives in Medical Simulation (a 501c6 trade organization—of which I am the Treasurer), the American College of Surgeons, and others might band together to promote this request. Commissioning such a study is within AHRQ’s purview, as they have previously requested policy studies from the Institute of Medicine (although perhaps not having the unique characteristics of a Decadal Survey). To make such a US Survey international would be difficult administratively. However, similar requests might be made to relevant oversight institutions in other countries or regions. Such Surveys could be done in parallel; or if necessary, a Survey completed in one country could be leveraged to trigger their conduct in other regions.
If the simulation community is successful in getting such Surveys commissioned and funded, we will need to mobilize substantial effort from our peers to provide the needed expertise and leadership. Although service on a Decadal Survey Committee is voluntary and typically carries no salary support, it is a highly prestigious activity. We should individually and collectively be willing to make the sacrifice necessary to see that Surveys can be completed successfully in all regions and then used to advance the understanding of and support for our field. Such intensive efforts will be worthwhile to steer the proper course for our investigations for the next decade and beyond.
1. Issenberg S, Ringsted C, Ostergaard D, Dieckmann P. Setting a research agenda for simulation-based healthcare education: a synthesis of the outcomes from an Utstein-style meeting. Simul Healthc 2011; 6: 155–167.
2. Dieckmann P, Phero J, Issenberg S, Kardong-Edgren S, Østergaard D, Ringsted C. The first research consensus summit of the Society for Simulation in Healthcare: conduction and a synthesis of the results. Simul Healthc 2011; 6: S1–S9.
3. Gaba DM. Where do we come from? What are we? Where are we going? Simul Healthc 2011; 6: 195–196.
4. Gaba DM. The future vision of simulation in health care. Qual Saf Health Care 2004; 13 (Suppl 1): i2–i10.
5. 5. Weiss J, Smythe WD, Lu W. Science traceability. 2005 IEEE Aerospace Conference, Big Sky, MT, March 2005.
6. 6. Gotel O, Finkelstein A. An analysis of the requirements traceability problem. Proceedings of the First IEEE International Conference on Requirements Engineering (ICRE ’94), 1994.
8. Fellows JD, Alexander JK. Decadal Science Strategy Surveys: Report of a Workshop. Washington, DC: National Academies Press; 2007. Available at: http://www.nap.edu/catalog/11894.html
. Accessed September 8, 2011.
9. Committee for a Decadal Survey
of Astronomy and Astrophysics; National Research Council. New Worlds, New Horizons in Astronomy and Astrophysics. Washington, DC: National Academies Press; 2010. Available at: http://www.nap.edu/catalog/12951.html
. Accessed September 8, 2011.
10. Committee on the Planetary Science Decadal Survey
; National Research Council. Vision and Voyages for Planetary Science in the Decade 2013–2022. Washington, DC: National Acadamies Press; 2011. Available at: http://www.nap.edu/catalog/13117.html
. Accessed September 8, 2011.
11. Panel on Implementing Recommendations from New Worlds, New Horizons Decadal Survey
; National Research Council. Report of the Panel on Implementing Recommendations from the New Worlds New Horizons Decadal Survey
. Washington, DC: National Academies Press; 2011. Available at: http://www.nap.edu/catalog.php?record_id=13045
. Accessed September 8, 2011.