Secondary Logo

Decision Support Using Anesthesia Information Management System Records and Accreditation Council for Graduate Medical Education Case Logs for Resident Operating Room Assignments

Wanderer, Jonathan P. MD, MPhil*; Charnin, Jonathan MD; Driscoll, William D.†‡; Bailin, Michael T. MD§; Baker, Keith MD, PhD

doi: 10.1213/ANE.0b013e318294fb64
Economics, Education, and Policy: Technical Communication
Free

Our goal in this study was to develop decision support systems for resident operating room (OR) assignments using anesthesia information management system (AIMS) records and Accreditation Council for Graduate Medical Education (ACGME) case logs and evaluate the implementations. We developed 2 Web-based systems: an ACGME case-log visualization tool, and Residents Helping in Navigating OR Scheduling (Rhinos), an interactive system that solicits OR assignment requests from residents and creates resident profiles. Resident profiles are snapshots of the cases and procedures each resident has done and were derived from AIMS records and ACGME case logs. A Rhinos pilot was performed for 6 weeks on 2 clinical services. One hundred sixty-five requests were entered and used in OR assignment decisions by a single attending anesthesiologist. Each request consisted of a rank ordered list of up to 3 ORs. Residents had access to detailed information about these cases including surgeon and patient name, age, procedure type, and admission status. Success rates at matching resident requests were determined by comparing requests with AIMS records. Of the 165 requests, 87 first-choice matches (52.7%), 27 second-choice matches (16.4%), and 8 third-choice matches (4.8%) were made. Forty-three requests were unmatched (26.1%). Thirty-nine first-choice requests overlapped (23.6%). Full implementation followed on 8 clinical services for 8 weeks. Seven hundred fifty-four requests were reviewed by 15 attending anesthesiologists, with 339 first-choice matches (45.0%), 122 second-choice matches (16.2%), 55 third-choice matches (7.3%), and 238 unmatched (31.5%). There were 279 overlapping first-choice requests (37.0%). The overall combined match success rate was 69.4%. Separately, we developed an ACGME case-log visualization tool that allows individual resident experiences to be compared against case minimums as well as resident peer groups. We conclude that it is feasible to use ACGME case-log data in decision support systems for informing resident OR assignments. Additional analysis will be necessary to assess the educational impact of these systems.

Published ahead of print June 7, 2013.

From the *Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee; Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital; Massachusetts General Hospital, Boston; and §Department of Anesthesiology, Baystate Medical Center, Springfield, Massachusetts.

Accepted for publication March 24, 2013.

Published ahead of print June 7, 2013.

Funding: Supported by Departmental funding.

The authors declare no conflict of interest.

This report was previously presented, in part, at the STA 2012 and at the American Society of Anesthesiologists Annual Meeting 2012.

Reprints will not be available from the authors.

Address correspondence to Jonathan P. Wanderer, MD, MPhil, Department of Anesthesiology, 1301 Medical Center Drive, Suite 4648, The Vanderbilt Clinic, Nashville, TN 37212. Address e-mail to jonathan.p.wanderer@vanderbilt.edu.

A fundamental challenge for graduate medical education in the United States is finding the right balance between service and education in an era of increasing concern over health care efficiency. For anesthesia residency programs, this issue manifests itself daily in the task of creating operating room (OR) assignments. Residents need exposure to a diverse set of cases that are appropriate for their level of training and meet their educational needs. In addition, case assignments must satisfy the minimum number of required case types set by the Accreditation Council for Graduate Medical Education (ACGME).1 Currently, residents log their case experiences manually through the ACGME Web site, but these data are not readily available in an actionable format to those responsible for making daily assignments. Furthermore, the granularity of the ACGME class classification schema is not sufficient to ensure that exposure to cases within all categories will satisfy educational objectives. For example, an awake craniotomy and an intraventricular shunt for hydrocephalus are both classified as intracerebral, nonvascular surgery. Although the ACGME has identified optimizing the ratio of service to education as a key priority, an unintended consequence of programs simply trying to ensure that minimum case numbers are met is that some residents may graduate without a sufficient breadth of experiences. Thus, programs aspiring to ensure such broad exposure must keep detailed resident case logs and make this information available when case assignments are made.

In the present system, the burden of ensuring that resident educational needs are met is placed on the shoulders of the anesthesiologists assigning cases, with periodic oversight by the program director. These objectives might be better served if responsibility for ensuring wide-ranging experiences were shifted to the residents. Learners demonstrate a moderate improvement in knowledge when using self-directed learning methods, compared with traditional methods.2 Additionally, studies from employee health literature suggest that giving workers choice and flexibility improves physical and mental health.3 To facilitate the process of making residents more responsible for ensuring that their personal educational needs are met, we developed a Web-based software program called Residents Helping in Navigating OR Scheduling (Rhinos). Rhinos solicits resident input on OR case assignments and creates resident profiles to facilitate attending review of case experience. Resident profiles are snapshots of the cases and procedures each resident has done and were derived from anesthesia information management system (AIMS) records and ACGME case logs. Separately, we developed an ACGME case-log visualization tool that allows individual resident experiences to be easily compared against case minimums as well as resident peer groups.

Back to Top | Article Outline

METHODS

This study was deemed exempt by the Partners HealthCare IRB (protocol 2011P000223) as it met criteria because of research conducted in an established educational setting. At the request of our residency program director (KB), the ACGMEa provided us with credentials for read-only access to the Resident Case-Log System via an ACGME Web service and supplied a reference document entitled “ACGME Web Service Developer’s Guide” that details the parameters required and values returned for each function call. An automated process was created that used the ACGME Web service to update a local copy of each resident’s ACGME case log every night from data entered within the previous 90 days. Resident demographic information was stored in a Microsoft Access (Microsoft Corporation, Redmond, WA) database and linked to ACGME case-log data. An ACGME case-log visualization tool was created using ASP.NET (Microsoft Corporation) that graphically displayed the number of logged procedures, required procedures, and peer-group (same Clinical Anesthesia-year) average procedures (Fig. 1A) for each category. Access to this application was available to members of the anesthesia department after authentication through the hospital intranet (accessible from any hospital computer or externally via virtual private network). Additionally, this tool permits exporting of data for further analysis (Fig. 1B).

Figure 1

Figure 1

Rhinos was composed of 2 main components: a graphical user interface developed in an Adobe Flash environment (Adobe Systems, San Jose, CA) and a server component that used Microsoft SQL Server (Microsoft Corporation) and VB.NET (Microsoft Corporation). The application was accessed via a Web link hosted within our AIMS. The default resident view was a case list for an upcoming clinical day specific to each resident’s current clinical service (Fig. 2A). Requests were made by dragging an OR from the case list to a request box, which permitted up to 3 ORs to be added along with an optional comment. The attending view was a list of requests and comments for the next business day with the ability to navigate each resident’s profile (Fig. 2B). Profile views include a summary of cases and procedures performed as derived from our AIMS (Fig. 2B), a list of monthly cases and ORs (Fig. 3, A and B) in addition to each resident’s ACGME data (Fig. 3C). A paging tool that provides the option of sending custom text pages to specific individuals was incorporated to facilitate the communication of finalized assignments. Residents were encouraged but not required to enter assignment requests each day. At our institution, residents are assigned to a specific clinical service each month and generally are assigned only to ORs for that service during the month. A service leader creates daily service schedules for 1 or 2 services. These portions of the assignment process were not modified.

Figure 2

Figure 2

Figure 3

Figure 3

The server component was implemented as an XML-based Web service that provided the user interface with a single point of contact for all backend functionality. A Microsoft SQL Server (Microsoft Corporation) database was used to store resident requests. Additionally, the Web service integrated with the OR scheduling system, the hospital paging system, our personnel scheduling system, and our MetaVision (iMDsoft, Needham, MA) AIMS database. Resident profiles were generated each night by an automated process that analyzed each case that each resident had performed and summarized local ACGME case-log data.

After developing Rhinos, a 6-week pilot project was started on 2 clinical services. Subsequently, Rhinos was implemented on all 8 of our OR-based anesthesia services for 8 weeks. Request data were then analyzed along with AIMS case records to determine the frequency of match success. Match success was defined as a resident performing at least one of the cases from the selected OR.

Back to Top | Article Outline

RESULTS

Pilot Project

Twenty-three residents placed assignment requests using the Rhinos system during the 6-week pilot period. The system was used on 2 separate clinical services by a single attending anesthesiologist who was responsible for creating resident and staff assignments for both services. A total of 165 assignment requests were entered. Of the 165 requests, 87 first-choice matches (52.7%), 27 second-choice matches (16.4%), and 8 third-choice matches (4.8%) were made. Forty-three requests were unmatched (26.1%) and 39 first-choice requests overlapped (23.6%).

Back to Top | Article Outline

Full Implementation

Sixty-four residents placed assignment requests using the Rhinos system during the 8-week full implementation period. The system was used on 8 separate clinical services. Fifteen attending anesthesiologists used the system to create assignments. A total of 754 assignment requests were entered. Of the 754 requests, 339 first-choice matches (45.0%), 122 second-choice matches (16.2%), and 55 third-choice matches (7.3%) were made. Two hundred thirty-eight requests were unmatched (31.5%). There were 279 overlapping first-choice requests (37.0%). Combining the pilot and full implementation requests resulted in an overall match success rate of 69.4%.

Back to Top | Article Outline

DISCUSSION

We successfully developed 2 decision support systems for resident OR assignments using ACGME case logs. Our ACGME case-log visualization tool provides a graphical view that allows easy comparison of each resident’s experience to both the ACGME case minimums and resident peer-group averages. Rhinos integrates multiple perioperative information systems with resident input in a manner that can support self-directed learning. Although there was the potential for residents to request identical assignments and thus create frequent conflict, in practice this occurred at a rate that was not problematic. Resident preferences varied sufficiently on a day-to-day basis to permit approximately half of first-choice assignment requests to be met. Both of these decision support systems provide ACGME case-log data in an easily usable format.

Automated analysis of resident-generated records has been described.4 Previous investigators have found that ACGME case logs and information derived from AIMS records deviates by at least 5% for the majority of residents.5 Given the current lack of an automated mechanism to resolve these discrepancies or eliminate them by allowing AIMS records to populate ACGME case logs, we included both sets of metrics within Rhinos’ resident profiles. With the current read-only ACGME case-log Web service, this is what is currently feasible. However, if a process were created to allow automated uploading of case data from AIMS into the ACGME case-log system, these discrepancies would likely be reduced and use of the case-log system would be more straightforward. Additionally, this would reduce resident documentation burden. We encourage the ACGME to provide enhancements to the existing Web service that would allow for full AIMS integration.

A key feature in the success of a system that uses resident input to inform resident assignments is the shared responsibility between residents and attendings for creating appropriate assignments. The traditional approach places the entire burden of the process on attendings and does not allow for consistent resident input. The alternate extreme would be to place the entire responsibility in the hands of residents, an approach that has been tried unsuccessfully elsewhere.6 The middle ground we have found using Rhinos, by contrast, has been successful.

The system that we have developed is made possible by our AIMS and electronic OR schedule, and required additional technical resources to implement. Analysis of resident case experience allows for more informed case assignments and can facilitate equitable distribution of cases. Combining these tools with resident input can lead to a successful implementation of a self-directed learning strategy. Increased autonomy is a well-described factor in increasing individual motivation, and motivation is a key component in learning.7 By soliciting input regarding upcoming clinical duties, residents may feel more engaged in and satisfied with the education process and have increased motivation to learn. Additionally, noneducational aspects of surgical scheduling may have also influenced resident choices. These factors may include desirable scheduled OR end times, the presence of large blocks of unscheduled OR time, the need to perform an inpatient preoperative evaluation, and an avoidance or preference for specific surgeons. As previously discussed, oversight and review of resident choices is crucial to successful implementation. Additional analysis will be necessary to determine the overall educational impact of this system.

Back to Top | Article Outline

DISCLOSURES

Name: Jonathan P. Wanderer, MD, MPhil.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Attestation: Jonathan P. Wanderer approved the final manuscript.

Name: Jonathan Charnin, MD.

Contribution: This author helped conduct the study.

Attestation: Jonathan Charnin approved the final manuscript.

Name: William D. Driscoll.

Contribution: This author helped conduct the study.

Attestation: William D. Driscoll approved the final manuscript.

Name: Michael T. Bailin, MD.

Contribution: This author helped conduct the study.

Attestation: Michael T. Bailin approved the final manuscript.

Name: Keith Baker, MD, PhD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Attestation: Keith Baker approved the final manuscript.

This manuscript was handled by: Edward C. Nemergut, MD.

Back to Top | Article Outline

FOOTNOTES

a Tom Richter (trichter@acgme.org), Director of Data Systems and Data Analysis, ACGME, 515 N. State St., Suite 200, Chicago, IL 60654.
Cited Here...

Back to Top | Article Outline

REFERENCES

1. . ACGME Program Requirements for Graduate Medical Education in Anesthesiology. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/040_anesthesiology_f07012011.pdf. Accessed June 11, 2012
2. Murad MH, Coto-Yglesias F, Varkey P, Prokop LJ, Murad AL. The effectiveness of self-directed learning in health professions education: a systematic review. Med Educ. 2010;44:1057–68
3. Joyce K, Pabayo R, Critchley JA, Bambra C. Flexible working conditions and their effects on employee health and wellbeing. Cochrane Database Syst Rev. 2010;2:CD008009
4. Bhattacharya P, Van Stavern R, Madhavan R. Automated data mining: an innovative and efficient web-based approach to maintaining resident case logs. J Grad Med Educ. 2010;2:566–70
5. Simpao A, Heitz JW, McNulty SE, Chekemian B, Brenn BR, Epstein RH. The design and implementation of an automated system for logging clinical experiences using an anesthesia information management system. Anesth Analg. 2011;112:422–9
6. Gross H, Kuhn W. Resident self-scheduling: a painful method of reinventing the wheel. Acad Emerg Med. 2000;7:208–9
7. Vansteenkiste M, Simons J, Lens W, Sheldon KM, Deci EL. Motivating learning, performance, and persistence: the synergistic effects of intrinsic goal contents and autonomy-supportive contexts. J Pers Soc Psychol. 2004;87:246–60
© 2013 International Anesthesia Research Society