Health care delivery systems trap rich clinical information, in the form of digitized free text1 and nonstandard nomenclature2,3 4,5 6 7
How will we provide information access to trainees as they assume the multiple roles of quality improvement (QI) specialist, clinician, and researcher while still protecting patient privacy? What software tool functionality must we create to enable health care providers to summarize outcomes over time across populations of patients and then identify specific patients who require intervention? The National Research Council stated recently that the absence of these integrative capabilities in existing information technology (IT) systems are responsible for an “IT health care chasm” and threaten the medical community's ability to meet the health care quality goals enumerated in Institute of Medicine reports.8
Montefiore Medical Center, the university hospital for the Albert Einstein College of Medicine, which is located in the Bronx, New York, had 93,752 discharges, 303,342 emergency room visits, and 1,970,920 visits to its ambulatory care service in 2009. To help track this vast amount of patient data, Montefiore has had an inpatient and evolving outpatient EMR since 1997. For over a decade, a small group of researcher–programmers has worked with the QI, research, operations, and education communities at Montefiore to ensure both meaningful access to data and the transformation of these data into actionable information. This report summarizes our experience creating software, Clinical Looking Glass (CLG), and incorporating information access into our health care delivery culture. Our successful approach to creating useful data and incorporating the resulting data into the organizational culture could serve as a potential national model.
For the purpose of this discussion, an EMR system includes, for each patient, a unique but universal patient identifier, a problem list, and structured laboratory, pharmacy, and ADT (admission, discharge, and transfer) data for both inpatient and outpatient care. An EMR may also include unstructured pathology, radiology, and nuclear medicine text reports. This system is an extremely fast filing cabinet optimized for displaying a single patient's data, but it comes with no guarantee that it can aggregate information for groups of patients (cohorts) in order to assess processes and outcomes across time.9
Trials and Tribulations
A decade ago (1998), our first attempt to obtain aggregate information from the EMR by direct query demonstrated the limitations of many EMR systems. We considered a simple QI question—about the adequacy of HIV care—that required information about the use of viral load testing. Within 15 seconds, our simple query requesting a list of patients with a viral load in the preceding month consumed 30% of the available computational power. A system administrator quickly aborted the query and revoked our rights to attempt another such search.
To avoid the load on the operational system, we replicated the EMR data elements (e.g., laboratory test results, number of hospital and clinic visits, and inpatient and outpatient prescriptions) in an industry-standard database. However, the nonintuitive table names and cryptic relationships demanded that the end user learn not only how to program analytic procedures for the database in the form of structured query language but also how to negotiate a data structure optimized for transactions, not for reporting. We found that commercially available, business-intelligence tools were geared for financial analysis. They allowed only a rigid notion of time linked to the calendar (e.g., quarterly or annual operations) and could not recognize the temporal evolution of outcome that permits each member of a patient cohort to have his or her own zero start time from which subsequent events should be analyzed.
Our third effort resulted in CLG—a proprietary, user-friendly software query system embedding core concepts of epidemiologic, longitudinal research in an accessible format. The tool, which is EMR brand agnostic (i.e., able to accept a wide variety of clinical event data from any system), helps users to define patient groups and analyze outcomes from within its vast clinical data repository. Because it searches for temporal relationships at the time of analysis, CLG allows clinicians to safely define their own queries directly without waiting for technology specialists to assist. Within a few hours of training, a doctor, nurse, administrator, or trainee has the ability in a nonidentified mode to qualify groups of patients or events, to establish a unique start time for each member of the group, to follow each individual's health state evolution over time (to death, readmission, or achievement of some targeted laboratory goal), and ultimately to make statistical comparisons of outcomes among groups. Those with privileged access can also identify specific patients in need of clinical remediation for outreach. The system has been in development for 10 years and in operation for 6 years.
Philosophy of CLG
While we were developing CLG, we worked to ensure that the system would subscribe to the following core principles:
The EMR must be properly engineered for efficient physician workflow encouraging physicians to enter clinical detail without feeling overburdened by unnecessary and disruptive data entry. The CLG tool must be able to produce meaningful outcomes and analyses of care quality from data collected in the normal course of patient care.10,11
Clinicians and authorized health care administrators must have direct access to data and to a query tool that enables real-time responses so as to encourage dynamic iterative exploration.
The tool must make epidemiologically sophisticated analyses easy to produce while also allowing idiosyncratic queries of unlimited complexity (e.g., it must accommodate multiple Boolean logic expressions [“and,” “or,” “not”] nested within parentheticals). This ability requires the statistical capacity to enable, for example, group comparisons, data visualization, and resampling for nonparametric analysis.
Patients must be qualified for inclusion in cohorts of interest not only by their static demographic descriptive properties but also by events that have temporal meaning in relation to some patient-specific start time (see “Patient-Centered Temporality” below). Recognizing these temporal relationships among a limitless diversity of clinical events is critical in identifying clinically relevant patient cohorts because both the act of qualifying patients for cohort inclusion and the act of identifying relevant analytic outcomes require temporally relating events at the time of analysis from the most current data. Because these question-specific relationships cannot always be predicted, preprocessing is not an option.12,13
Users must be able to upload patient-group data that exist outside of the tool for further analysis and comparison with other groups.
Output must be self-documenting to ensure reproducibility. This means that each report or data output must include a complete description of the criteria and sources used to create it. The creation of this documentation should be automatic when the user submits his or her analysis for execution.
Output must be in at least four forms:
Reports for easy redistribution to clinicians or administrators with responsibility for oversight and interventional remediation of care (with or without patient identifiers as required for the purpose envisioned).
Data sets to allow further statistical analyses outside of the tool. Researchers or QI experts might wish to apply to the data new statistical or analytic techniques not yet available within CLG. To permit this, CLG must facilitate the export of data elements for absorption into external statistics packages.
Cohort objects that can become the input of additional analyses in other modules; that is, the output from one analysis must be able to become the input for the next, allowing for serial enrichment of the analytic question. For example, after completing an analysis on diabetic patients, a physician may observe that one group has gone on to good control and another failed to achieve targeted goals. He or she must, as a next step, be able to take the output of the first analysis—the groups with different outcomes—and ask a second question (e.g., what was the subsequent hospitalization experience of those who did [or did not] achieve target goals of diabetes control?).
Interactive cross-tabulation pivot tables (i.e., “data cubes”) for slice-and-dice analysis. One simple example is a table that shows average lab value for patients by both gender and ethnicity.
Health Insurance Portability and Accountability Act (HIPAA) protection must be built into the application (see “HIPAA and Privacy” below). Likewise, the application should include a controlling policy and standard operating procedures. CLG recognizes the moral and legal imperative incumbent on health care delivery systems to constantly review their practices and outcomes to ensure patients' quality of care. For QI activity at Montefiore, no patient consent is required, but full patient identifiers are available for QI investigators who need them. However, CLG at Montefiore requires the enrollment of the QI project with the Office of Performance Improvement before identifiers are released. For research, Montefiore requires institutional review board (IRB) oversight and permission, and the IRB makes decisions regarding waiver of requirement for consent on a case-by-case basis. In addition, CLG can perform its core functions in a restricted mode so the user does not require identifiers. In those situations in which the user does require the identifiers, the IRB makes the determination as to waiver of consent, and all identified analyses are auditable and include an automatically produced permanent record of the query and the person who made it.
Patient-Centered Temporality
To better understand the importance of patient-centric time frames of analysis, consider a cohort of 10 hypothetical newly diagnosed diabetic patients (represented in Figure 1A ). All initiated therapy in the year 2005, but some earlier in the year and some later. We follow them forward in time over the course of the year until they achieve the targeted hemoglobin A1c (HbA1c) value. From a calendar-centric point of view, only four (40%) of this cohort of patients achieve the targeted end point by the end of the year. The problem with this view is that each patient does not have an equal amount of time for the intervention to take hold. In fact, a doctor accepting a patient in November would have only two months to bring the patient's diabetes under control—a clear and perverse disincentive to accept this patient for care. CLG allows a patient-centric view with a unique start time for each patient (as represented in Figure 1B ). CLG sorts the patients by amount of follow-up time, from those requiring the least amount of time to achieve the targeted HbA1c value to those requiring the greatest. The scale is no longer the calendar but, rather, the time elapsed from each patient's unique start of treatment. The question is no longer the limited, “Which patients are in control in 2005?” but, rather, the more meaningful accountability question, “What percent of patients who were enrolled in 2005 and followed for a year were brought into control?” The results are now 5 of 10, or 50%. Recognizing temporal relationships, that is, respecting each patient's unique start time, is critical for meaningful analysis, and the ability to do so represents a major advantage of CLG over calendar-centric analytic tools. This innovative software allows researchers to identify cohorts in a profoundly new way, and this cohort paradigm allows them to conduct epidemiologically interesting analyses that are focused on the experience of individual patients within a group (e.g., amount of time in therapeutic intervention) and to qualify patients on the basis of multiple sequential events.
Figure 1:
The greater fidelity of time-based queries in Clinical Looking Glass, software developed and used at Montefiore Medical Center of the Albert Einstein School of Medicine in the Bronx, New York, as compared with time-based queries in traditional databases. Calendar-centric measurement (A) produces different results from patient-centric measurement (B) when using the same patient data. The patient number shows the different sort order between the two perspectives.
To help a user define a patient cohort using temporal relationships and infinite combinations of Boolean operators, a simple criteria entry form did not suffice. Instead, CLG uses an “Event Canvas” on which the user builds the group definition. Event Canvas allows a specific group definition (see Figure 2 ), such as a cohort of acute myocardial infarction (AMI) patients who may have suffered the effects of a negative drug interaction between the antiplatelet agent (clopidogrel) and proton pump inhibitors (PPI, which is given for gastric acid control but is also known to functionally inactivate the effect of clopidogrel). Writing another line in Event Canvas allows the clinician to identify all AMI patients in a given time frame (e.g., years 2004–2008). A third line can help the clinician further narrow the search to patients who began one particular treatment (e.g., a clopidogrel prescription started within 30 days of AMI discharge), and another line will allow him or her to locate patients within that group who were also prescribed another treatment (in this case, a prescription of PPI, the inactivating medication, to start within 30–365 days after the clopidogrel). The top line in Event Canvas isolates the earliest start date (the “index date”) of the group with the additional medication (PPI) to represent the patient cohort in further analysis. For example, a clinician can now compare this patient cohort with another just like it who did not undergo the same treatment (patients who, in the example, do not have PPI prescriptions) in order to assess outcomes such as mortality and new onset of heart attack.
Figure 2.:
A partial screenshot depicting a patient-group definition, including the temporal relationship between events, in Event Canvas, the query tool of Clinical Looking Glass, software developed and used at Montefiore Medical Center of the Albert Einstein School of Medicine in the Bronx, New York.
To study outcomes for a given patient cohort, CLG users can define similar temporal relationships to query for outcome events relative to the subject-specific index date. Depending on his or her analysis needs, the user can query outcomes “before,” “after,” and “around” the cohort index date. CLG also permits users to define their own durations based on clinical events of individual patients and then to query the outcomes of those analyses; for example, they may analyze INR results (which measures clotting tendency in the blood) to determine the time between discharge for atrial fibrillation and a first major hemorrhage.
Outcome analysis can involve either simple methods, such as counting, or more sophisticated methods, such as evaluating the amount of time that patients' lab values remain in user-defined categories of laboratory value ranges. Thus, determining the time a patient maintained a category of good INR (defined as an “INR between 2 and 3 in the year of follow-up postfirst-discharge with atrial fibrillation”) is a doable task. Additionally, an embedded statistical service produces time to outcome event, cumulative percentage attaining outcome, and Kaplan–Meier statistics for censored data with model point estimates, confidence intervals, and P values for group comparison.
CLG Usage at Montefiore
When we wrote this report, over 700 clinicians, researchers, administrators, and managers had received CLG training, and more than 500 were then active users. In addition, for the last three years, all medical residents have received training in CLG during the ambulatory care month of their first year. At the end of the training, they must demonstrate sufficient proficiency to ask and answer a clinical question, within an area of their own interests, using CLG. Several clinicians and clinicians-in-training have used CLG to prepare for and/or perform studies that have led to peer-reviewed publications which summarize QI efforts, report research findings, and assess the impact of national policy changes in rehabilitation medicine reimbursement on readmission and mortality.6,7,14–27
Comparative effectiveness
A chief resident in ambulatory care collaborated with her fellow residents to undertake a QI project to compare the effectiveness of an intensive diabetes care program for chronically uncontrolled patients with that of standard care in other settings. She used CLG to obtain data that demonstrated a mean absolute HbA1c reduction of 1.49 in six months in the intensive care program versus 0.69 in the comparison group.
Laboratory quality assurance
In 2007, Montefiore endocrinologists were referred a number of patients for hypercalcemia and presumably hyperparathyroidism who, after extensive evaluation, were found to be clinically normal. Standard quality control samples failed to detect a laboratory problem. Thirty days later, a medical director reported an epidemic of elevated calcium. The laboratory reassured him that all was well. The medical director, however, having completed training in CLG, built cohorts of patients in different months and demonstrated a dramatic increase in patients with calcium greater than 10.5 mg/dL right above the boundary of normal. Faced with this evidence, the laboratory realized that its control for hypercalcemia focused on the more extreme edge of the calcium detection range. Using a secondary control standard, the laboratory demonstrated a consistent calcium drift upward of 0.4 mg/dL in the range of the clinical decision point. CLG creators never anticipated this use; however, a clever clinician faced with a real informational need used the capacities of CLG to halt a “faux epidemic” and its attendant cost in dollars and patient anxiety.
Adverse drug reaction
Gatifloxacin had been on the market for six years before a definitive epidemiology study in the New England Journal of Medicine (NEJM ) demonstrated its dysglycemic toxicity in 2006.28 29 NEJM editors were pining.
Support for grant-seeking
Academic medical centers survive on their ability to obtain and support grants. Fellows involved in the following programs use CLG to identify cohorts of patients to evaluate program success through built-in CLG metrics:
the National Institutes of Health (NIH)-funded Clinical Research Training Program at Albert Einstein College of Medicine,
the Center for AIDS Research at the Albert Einstein College of Medicine,
the Reynold's Foundation grant program in geriatric education, and
the Robert Wood Johnson Foundation's “Expecting Success: Excellence in Cardiac Care Program.”
As NIH strategically focuses on translational medicine, organizations that have the appropriate intellectual ecosystem for efficient translational research and that are capable of coordinating research and clinical assets will have a competitive advantage. CLG capabilities were important to the successful application by Einstein and Montefiore for an NIH Clinical and Translational Science Award.
Knowledge sharing
In many institutions, negotiating institutional data assets is a lonely and often selfish endeavor. Each researcher extracts information for his or her own use without contributing to the data knowledge of the academic community. This is not merely a reflection of academic narcissism; currently, no processes or mechanisms allow general sharing of an individual's experience of the institution's data assets. CLG corrects this deficit. A common environment for research, QI, and exploration transforms every study performed by any academic into a quality check of the data and its analytic capabilities. Errors found in data or in software are corrected immediately and benefit all colleagues without necessitating personal consultations. A common wiki permits knowledge to be generally shared. Bimonthly CLG research meetings held in the Division of General Internal Medicine provide opportunities to present research in evolution, supporting the research community's use of CLG.
Public health use
The potential public health utility of CLG is clear, as demonstrated for postmarketing surveillance by the gatifloxacin example cited above. Further, broad reviews of the potential public health utility of medical informatics indicate the need for a tool like CLG.30
Taxonomy of Organizational Access to Clinical Information
To properly contextualize the CLG information access model at Montefiore, it is helpful to describe the spectrum of health care organization approaches to information creation and access and to explain why Montefiore chose the approach it uses with CLG.
Priesthood
Most organizations empower a small executive group with access to information, that is, a “priesthood” of analysts who learn to extract data from a variety of sources using technically sophisticated tools and programming skills. We liken this group of analysts to a body of ordained religious practitioners because of the special knowledge they must possess to do their jobs. This approach is expensive, rations access to the most powerful, and denies the organization the benefit of the creativity and intelligence of the clinical workforce who have no expectation of access.
Command and control
The second approach permits widespread distribution of select reports whose content is determined by an expert group. A good example of this approach is a dashboard of performance metrics. The workforce is expected to react to the predefined report, but it may neither define nor modify the report. A variant of this approach is the parameter-driven report, to which the workforce is able to make only minor modifications, such as date ranges, at run time.
Data marts
The third approach is to provide subject-specific repositories of data (data marts) to meet the reporting needs of a well-funded clinical department or research community. An administrator organizes the data to answer typical questions of the specific user-community and often applies definitional rules (such as the criteria for a positive diagnosis of diabetes) to create the data mart, making changes to the criteria expensive and difficult.
Data democracy
This is the approach that we are pursuing at Montefiore with CLG. It assumes that patients and organizations benefit when the health care workforce can easily bring its creative energies and intellect to bear on the phenomena of health care delivery. The ability of users to access all data using well-defined and easily used analytic patterns to answer unexpected questions without requiring preauthorization or special additional resources makes this approach the most cost-effective. Such open and inexpensive access should be widely available to clinicians and administrators committed to continuous QI.31
Democratization of Access—Idiosyncratic Montefiore Goal or National Aspiration?
A review of publicly stated policies of national bodies foreshadows the formal requirement for the sort of access that Montefiore has already enabled and already offers its clinicians, researchers, educators, and QI investigators.
The American Board of Internal Medicine32
The Accreditation Council for Graduate Medical Education33
The 2009 standards of the Joint Commission on Accreditation of Health Care Organizations34
The value-based purchasing concept of the Center for Medicare and Medicaid Services' links payment directly to the quality of care provided.35
HIPAA and Privacy
Enabling this widespread access while still protecting patient privacy required thoughtful dynamic codevelopment of software and organizational culture.36 37 31
On the software side, CLG performs all analyses in the restricted mode—that is, deidentified—by default. The output of such a mode, while still, according to HIPAA, a “limited data set” (as it includes dates of birth and dates of service), excludes names, medical record numbers, addresses, or geocoding below the level of census tract. We formally train users to treat such limited data sets with great care. The Montefiore IRB ruled that use of CLG in the restricted mode under specific restrictions does not require preapproval from the IRB, which enables a large number of trainees and clinicians to use CLG to explore project feasibility before committing to an IRB submission.
All use of CLG in the privileged mode (with identifiers) requires the user to explicitly demand access despite a warning to try to avoid using identifiers. Only a contemporaneous declaration of the authority under which the user is accessing identified data can override the system, and the system then records this declaration. Currently, users can claim one of three authorities: QI, IRB-approved research, or patient work list. All QI projects are preregistered with the Office of Performance Improvement in a tracking system that provides CLG with the names of all active QI projects so that users can select their QI project from a dropdown list. CLG requires that IRB projects be named at runtime. Finally, users need a description of intent to obtain an identified patient work list. When user queries produce patient identifiers, CLG records for future audit the user, the time, the justification, and the query. The IRB and Office of Performance Improvement can review at will the users and the justifications of use through a special report CLG can create for them. A formal audit of queries and uses is conducted annually.
Call to Action
Democratization of meaningful access to clinical data for QI, IRB-approved research, and clinician education is within our technical capability and is clearly (as shown above) an aspiration of the policy-making bodies in the country. The Montefiore experience demonstrates what is possible, and it is imperative that the United Sates develop the technical, legal, and political processes that can bring this valuable technology to all clinicians, medical researchers, medical educators, and medical trainees. Democratizing data will not be easy. What has been accomplished at Montefiore will require a national health identifier (to link all clinical records), a well-developed law to protect privacy and to severely penalize those who would abuse the resource, and the political will to create incentives (such as clinician and institutional reimbursements) for using a tool like CLG to improve patient care. Although these needs are great, there is a growing consensus that health care delivery systems must modernize using IT to make health care efficient, affordable, and safe. The academic medicine community must take the lead in introducing this powerful paradigm to trainees so that they are prepared for a future of engaged creativity and shared longitudinal accountability for outcome.
Acknowledgments:
The authors would like to acknowledge the hard work of dozens of colleagues at Montefiore Medical Center and the Albert Einstein College of Medicine who have used, supported, and developed Clinical Looking Glass.
Funding/Support:
None.
Other disclosures:
None.
Ethical approval:
All clinical work in Clinical Looking Glass is approved through exempt quality improvement activity overseen by the Office of Performance Improvement of Montefiore Medical Center or through specific research protocols of the Montefiore Institutional Review Board.
References
1 Schleyer T. A salient problem in informatics? J Am Med Inform Assoc. 2008;15:707.
2 Kannry JL, Wright L, Shifman M, Silverstein S, Miller PL. Portability issues for a structured clinical vocabulary: Mapping from Yale to the Columbia medical entities dictionary. J Am Med Inform Assoc. 1996;3:66–78.
3 Bouhaddou O, Warnekar P, Parrish F, et al. Exchange of computable patient data between the Department of Veterans Affairs (VA) and the Department of Defense (DoD): Terminology mediation strategy. J Am Med Inform Assoc. 2008;15:174–183.
4 Grant A, Moshyk A, Diab H, et al. Integrating feedback from a clinical data warehouse into practice organisation. Int J Med Inform. 2006;75:232–239.
5 Gray GW. Challenges of building clinical data analysis solutions. J Crit Care. 2004;19:264–270.
6 Reznik M, Hailpern SM, Ozuah PO. Predictors of early hospital readmission for asthma among inner-city children. J Asthma. 2006;43:37–40.
7 Segal M, Pedersen AL, Freeman K, Fast A. Medicare's new restrictions on rehabilitation admissions: Impact on the elderly. Am J Phys Med Rehabil. 2008;87:872–882.
8 Stead W, Lin H; Committee on Engaging the Computer Science Research Community in Health Care Informatics. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. Washington, DC: The National Academies Press; 2009.
9 Lau B, Gange SJ, Moore RD. Interval and clinical cohort studies: Epidemiological issues. AIDS Res Hum Retroviruses. 2007;23:769–776.
10 Wisniewski MF, Kieszkowski P, Zagorski BM, Trick WE, Sommers M, Weinstein RA. Development of a clinical data warehouse for hospital infection control. J Am Med Inform Assoc. 2003;10:454–462.
11 Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: Implications for performance measures. J Am Med Inform Assoc. 2007;14:10–15.
12 Zielstorff RD. Capturing and using clinical outcome data: Implications for information systems design. J Am Med Inform Assoc. 1995;2:191–196.
13 Dolin RH. Modeling the temporal complexities of symptoms. J Am Med Inform Assoc. 1995;2:323–331.
14 Abdu A, Gómez-Márquez J, Aldrich TK. The oxygen affinity of sickle hemoglobin. Respir Physiol Neurobiol. 2008;16:92–94.
15 Billett HH, Jacobs LG, Madsen EM, Giannattasio ER, Mahesh S, Cohen HW. Efficacy of inferior vena cava filters in anticoagulated patients. J Thromb Haemost. 2007;5:1848–1853.
16 Bonuck KA, Trombley M, Freeman K, McKee D. Randomized, controlled trial of a prenatal and postnatal lactation consultant intervention on duration and intensity of breastfeeding up to 12 months. Pediatrics. 2005;116:1413–1426.
17 Bonuck KA, Freeman K, Trombley M. Randomized controlled trial of a prenatal and postnatal lactation consultant intervention on infant health care use. Arch Pediatr Adolesc Med. 2006;160:953–960.
18 Boucai L, Surks MI. Reference limits of serum TSH and free T4 are significantly influenced by race and age in an urban outpatient practice of medicine. Clin Endocrinol (Oxf). 2009;70:788–793.
19 Curzio KM, Cheng-Lai A, Kheyfets V, Sinnet M, Billett HH. A comparison of direct thrombin inhibitors in the treatment of heparin-induced thrombocytopenia: A single institution experience. J Thromb Thrombolysis. 2009;28:117–123.
20 Ehrlich AR, Bak RY, Wald-Cagan P, Greenberg DF. Risk factors for fires and burns in homebound, urban elderly. J Burn Care Res. 2008;29:985–987.
21 Freeman K, Bonuck KA, Trombley M. Breastfeeding and infant illness in low-income, minority women: A prospective cohort study of the dose–response relationship. J Hum Lact. 2008;24:14–22.
22 Kamran M, Wachs J, Putterman C. Necrotizing fasciitis in systemic lupus erythematosus. Semin Arthritis Rheum. 2008;37:236–242.
23 Keene A, Ashton L, Shure D, Napoleone D, Katyal C, Bellin E. Mortality before and after initiation of a computerized physician order entry system in a critically ill pediatric population. Pediatr Crit Care Med. 2007;8:268–271.
24 Keene A, Kawano T, Anees S, Chen J. Increased bleeding risk associated with the use of recombinant human activated protein C in patients with advanced liver disease. Crit Care. 2008;12:405.
25 Kunins HV, Bellin E, Chazotte C, Du E, Arnsten JH. The effect of race on provider decisions to test for illicit drug use in the peripartum setting. J Womens Health. 2007;16:245–255.
26 Shuter J, Sarlo JA, Kanmaz TJ, Rode RA, Zingman BS. HIV-infected patients receiving lopinavir/ritonavir-based antiretroviral therapy achieve high rates of virologic suppression despite adherence rates less than 95%. J Acquir Immune Defic Syndr. 2007;45:4–8.
27 Southern WN, Berger MA, Bellin EY, Hailpern SM, Arnsten JH. Hospitalist care and length of stay in patients requiring complex discharge planning and close clinical monitoring. Arch Intern Med. 2007;167:1869–1874.
28 Park-Wyllie LY, Juurlink DN, Kopp A, et al. Outpatient gatifloxacin therapy and dysglycemia in older adults. N Engl J Med. 2006;354:1352–1361.
29 Gurwitz JH. Serious adverse drug effects—Seeing the trees through the forest. N Engl J Med. 2006;354:1413–1415.
30 Koo D, O'Carroll P, LaVenture M. Public Health 101 for informaticians. J Am Med Inform Assoc. 2001;8:585–597.
31 Bellin E, Dubler NN. The quality improvement–research divide and the need for external oversight. Am J Public Health. 2001;91:1512–1517.
32 American Board of Internal Medicine. ABIM Maintenance of Certification. Available at:
http://www.abim.org/moc . Accessed March 11, 2010.
33 Accreditation Council for Graduate Medical Education. Common Program Requirements. Available at:
http://www.acgme.org/acWebsite/dutyHours/dh_dutyhoursCommonPR07012007.pdf . Accessed March 11, 2010.
34 Joint Commission on Accreditation of Healthcare Organizations: 2009 Accreditation Requirements: Pre-Publication Version. Oakbrook Terrace, Ill: Joint Commission of Accreditation of Healthcare Organizations; 2008.
35 CMS Hospital Pay-for-Performance Workgroup; RAND Corporation; Brandeis University; Booz Allen Hamilton; Boston University. U.S. Department of Health and Human Services Medicare Hospital Value-Based Purchasing Plan Development. Available at:
http://www.cms.hhs.gov/AcuteInpatientPPS/downloads/hospital_VBP_plan_issues_paper.pdf . Accessed March 11, 2010.
36 Aarts J, Doorewaard H, Berg M. Understanding implementation: The case of a computerized physician order entry system in a large Dutch university medical center. J Am Med Inform Assoc. 2004;11:207–216.
37 Bloomrosen M, Detmer D. Advancing the framework: Use of health data—A report of a working conference of the American Medical Informatics Association. J Am Med Inform Assoc. 2008;15:715–722.
