Share this article on:

Comparative Effectiveness Research in DARTNet Primary Care Practices: Point of Care Data Collection on Hypoglycemia and Over-the-Counter and Herbal Use Among Patients Diagnosed With Diabetes

Libby, Anne M. PhD*; Pace, Wilson MD; Bryan, Cathy MHA, BSN, RN; Orton Anderson, Heather PhD*; Ellis, Samuel L. PharmD*; Read Allen, Richard MS§; Brandt, Elias BS; Huebschmann, Amy G. MD; West, David PhD; Valuck, Robert J. PhD, RPh*

doi: 10.1097/MLR.0b013e3181ddc7b0
Comparative Effectiveness

Background: The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a federated network of electronic health record (EHR) data, designed as a platform for next-generation comparative effectiveness research in real-world settings. DARTNet links information from nonintegrated primary care clinics that use EHRs to deliver ambulatory care to overcome limitations with traditional observational research.

Objective: Test the ability to conduct a remote, electronic point of care study in DARTNet practices by prompting clinic staff to obtain specific information during a patient encounter.

Research Design: Prospective survey of patients identified through queries of clinical data repositories in federated network organizations. On patient visit, survey is triggered and data are relinked to the EHR, de-identified, and copied for evaluation.

Subjects: Adult patients diagnosed with diabetes mellitus that scheduled a clinic visit for any reason in a 2-week period in DARTNet primary care practices.

Measures: Survey on hypoglycemic events (past month) and over-the-counter and herbal supplement use.

Results: DARTNet facilitated point of care data collection triggered by an electronic prompt for additional information at a patient visit. More than one-third of respondents (33% response rate) reported either mild (45%) or severe hypoglycemic events (5%) in the month before the survey; only 3 of those were also coded using the ICD-9 (a significant difference in detection rates 37% vs. 1%). Nearly one-quarter of patients reported taking an OTC/herbal, 4% specifically for the treatment of symptoms of diabetes.

Conclusions: Prospective data collection is feasible in DARTNet and can enable comparative effectiveness and safety research.

From the *University of Colorado School of Pharmacy, Denver, CO; †University of Colorado School of Medicine, Denver, CO; ‡American Academy of Family Physicians, Denver, CO; §Peak Statistical Services, Evergreen, CO; and ¶QED Clinical Inc., dba CINA, Dallas, TX.

Reprints: Anne M. Libby, PhD, 12631 E. 17th Ave., C238-L15, Academic Office 1, University of Colorado School of Pharmacy, Aurora, Colorado. E-mail:

The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a federated network of electronic health data designed to facilitate observational comparative effectiveness research and examine outcomes associated with medical care, including prescription medications and devices.1 Since Autumn 2009, DARTNet links patient-level clinical data from 19 organizations that are organizationally and geographically distinct, representing ∼100 primary care practices, 800 clinicians, and 1 million patients. At the time reported on in this manuscript, DARTNet included >500 clinicians and 400,000 patients. This article presents the contributions of DARTNet to the field of comparative effectiveness and safety research by focusing on the ability of DARTNet to facilitate data collection at the point of care to enhance clinical data for outcomes analysis.

DARTNet was developed to help study important clinical questions in ways that have not before been easily possible.2 Conventionally, observational comparative effectiveness research has been conducted using datasets created for other purposes, such as insurance claims data. Cohorts of patients created from large populations are then compared. Statistical matching algorithms are applied to try to account for underlying clinical differences among the groups, and the resulting analysis provides comparisons of outcomes of treatment. Although this methodology is powerful, it has been criticized because it cannot account for important clinical information that is not available in claims databases.3 Ultimately, even the best adjustment cannot adequately substitute for the addition of superior, existing clinical data or the collection of new data at the point of care.4 For example, in claims database studies of depression, measures of both severity and clinical response are lacking and in studies of diabetes mellitus, accurate reporting of hypoglycemic event rates and over-the-counter (OTC) product use are typically not available.5 DARTNet provides a vehicle by which such data can be obtained, either from electronic health records (EHRs) or from point-of-care data collection.

This article contributes to the advancement of comparative effectiveness research methods by illustrating how point-of-care data collection can contribute to improved detection of important outcome measures (mild and severe hypoglycemic events) and confounders (OTC product use) in the case of Type II diabetes therapy. It adds to our knowledge base by providing estimates of event rates that can be refined by larger scale point of care data collection. This “proof of concept” work is important in establishing both the use of DARTNet itself and the merit of linking data from multiple sources to improve measurement, statistical power, and the clinical validity of research findings.

Back to Top | Article Outline


Study Setting: DARTNet Nonintegrated Primary Care Practices

Through its Developing Evidence to Inform Decisions about Effectiveness Network, the Agency for Healthcare Research and Quality funded the development of DARTNet as a platform for next-generation comparative effectiveness research on medication and medical treatments in real-world settings. A program of the Colorado Developing Evidence to Inform Decisions about Effectiveness Center since 2007, DARTNet was developed as a federated network of geographically dispersed, electronic health record-enabled health care organizations for the purposes of research and the creation of a learning community to improve practice. DARTNet links information from nonintegrated primary care clinics that use EHRs to deliver ambulatory care. The system captures, codifies, and standardizes >120 unique data elements per patient for 48 months or more of time. It is a bidirectional exchange platform that permits prospective data collection and enhanced clinical data in usual care settings. DARTNet has been described in more detail elsewhere.1

The initial DARTNet prototype study was on oral hypoglycemic medications for adult patients diagnosed with diabetes mellitus. A current expansion study examines antidepressant treatment of adolescents and adults with depression. Both diabetes and depression efforts focused on validation studies comparing large-scale observational data to electronic health data from DARTNet providers; both efforts recruited primary care and multispecialty practices to participate in practice-based research and in a learning community for improving patient care; and both efforts included funded and unfunded pilot studies of point-of-care data collection in the remote practice sites triggered by electronic records.

Table 1 describes a patient population in DARTNet practices that had at least 24 months of continuous records at the beginning of 2009 (the available patient population expands as the time span requirement is reduced). More than 335,000 patients were enrolled with DARTNet primary care providers for their health care, and more than 446,000 were enrolled in primary or specialty care practices. Prevalence rates of diagnosed diabetes mellitus and depression, consistent with community studies, are roughly 10% of the total enrolled patient population. As expected, a higher proportion of the age 60 and older population (16%) account for (type 2) diabetes cases; depression is more evenly spread among adult subpopulations (11%–13%). Since October 2009, DARTNet included 19 organizations representing over 800 clinicians and 1 million patients.



Back to Top | Article Outline

Procedures: Point-of-Care Prospective Data Collection

This manuscript reports on a point of care study of patients with diabetes mellitus who were surveyed about the prevalence of hypoglycemic events and their use of OTC medications and herbal supplements. This was undertaken to test the feasibility of obtaining point-of-care data prospectively in DARTNet practices; this procedure is an electronic, EHR-based alternative to the card study procedures common to practice-based research networks. Human subjects’ approval was obtained for this unfunded pilot study from the University of Colorado, Denver Institutional Review Board (COMIRB 08-0202) and from the American Academy of Family Physicians IRB (08-053).

The researchers presented the pilot study concept to the DARTNet Oversight Committee and asked for volunteers to either pilot the point-of-care data collection themselves or recruit 1 or 2 clinicians from within their organizations. The researchers also recruited 1 DARTNet organization that is not represented on the oversight committee. The oversight committee recruited the participation of DARTNet clinicians with a standardized email and with additional follow-up emails or calls only to answer questions. The participation window was for a 2-week span from initiation in the practice from January to June 2009. Most DARTNet organizations agreed to participate. Data collection was completed in ∼2 weeks from initiation because total observations were ample for the pilot study (286 patients surveys completed). Because the target sample size was obtained, no further recruitment efforts were made for particular practices or for particular subpopulations.

A new protocol was integrated into a commercial clinical decision support system (QED Clinical, Inc., dba CINA) operated in all current DARTNet practices.6 CINA identified adult patients with a diagnosis of diabetes mellitus who were scheduled to visit the clinic and printed the point of care (POC) survey instrument along with the patients’ regular POC clinical decision support report. The survey instrument contained the patients’ name, visit date, and provider name. The staff would present the survey to the patient for completion during the visit (typically by the front office staff or the rooming nurse.) The patient would return it in any degree of completion to the staff. The staff would then fax the survey to CINA, a HIPAA business partner with each of the DARTNet organizations. CINA would enter the data into the clinical data repository for that record, and then de-identify the data before transferring a copy of the clinical data record to DARTNet for study. In this way, patient data and personal health records stay within the confines of the practices. DARTNet researchers obtain de-indentified copies of linked survey and electronic health data through a secure transfer process; they have data use agreements with practices and IRB (Institutional Review Board for the protection for human subjects) approval, and when the EHR records are copied, they are de-identified before transfer to researchers. This procedure is possible in virtually any EHR platform because it relies on open database connectivity compliance for standard conversion that has been the industry standard for a decade (Wilson Pace, personal communication, June 2009).

CINA also kept a daily log of the reports on diabetes patients eligible for the surveys that were generated; 872 surveys were identified this way and distributed across DARTNet practices. CINA also tallied surveys faxed to them for data entry and checked them off against the logged reports. From this tracking database, there were 329 reports returned (463 not returned). Of these 329, forty reports were returned but marked as “not complete,” meaning there was nothing completed on the form at all. Reasons for this are stated because the patient did not show up for the scheduled appointment; patient seen by a different provider; and some with no reason given. Three were not matched to unique EHR data, which could indicate duplicate surveys for 3 patients. There were surveys and linked EHR data for 286 patients, yielding a return rate of 38% and a response rate of 33% with wide variation by organization and practice.

The result is a pilot of prospective data collection, triggered by CINA's clinical decision support system programmed to prompt targeted surveys, conducted at the point of care; survey results are linked to electronic clinical and billing records remotely and electronically. Figure 1 displays the survey instrument on hypoglycemic events and OTCs/herbal supplements use.



Back to Top | Article Outline

Measures and Analysis

Hypoglycemic events were measured in the survey by 2 questions intended to approximate mild or severe events in the past month using standard definitions of severity.5,7 Mild was asked as needing “to eat or drink something because you felt your blood sugar was too low” (Question 1). Severe hypoglycemic event was asked as “has your blood sugar fallen so low that you required assistance from someone else” (Question 2). Hypoglycemia prevalence and incidence rates were reported, per standard conventions.7 Diagnosis codes of hypoglycemia (ICD-9 codes 251.0, 251.1, 251.2, 250.8, 250.80, 250.82) or a glucose laboratory test <70 mg/dL for hypoglycemic events during that same period for each patient were used as a medically coded measure. A McNemar test of differences for dependent data was used to compare the proportion of patients with hypoglycemia detected from each source. Statistical testing was not used to compare event rates because these event rates were not detectable in the medically coded measures used.

OTC or herbal supplement use was measured using a prompted list for any used in the past month: Ginko Biloba; chromium; cinnamon or cinnamon extract; garlic extract; gymnema sylvestre; fenugreek; ginseng; vanadium/vanadyl/panchromium; vitamin E, not in a multivitamin; and magnesium. Clinical experts (Ellis, Pace) and the literature were used to guide the selection of OTCs/herbals on the list. For example, cinnamon extract was shown in an Randomized Controlled Trial (RCT) to lower hemoglobin A1C 0.83%, considered both clinically and statistically significant.8 An open-ended question was then asked about OTC medications or herbs used specifically for diabetes. Answers to both queries were combined into OTC/herbal use.

Back to Top | Article Outline


Table 2 presents survey results from 286 patients about their experience of hypoglycemia events. Nearly one-fourth of patients reported at least 1 mild event in the month before their visit; 12% reported 3 or more. Five percent (n = 15) reported at least 1 severe hypoglycemic event. Hypoglycemia event incidence rates were 6.8 and 0.6 events per patient per year for mild and severe events, respectively, calculated as [(33 patients × 3 events) + (65 patients × 1 event)]/286 patients × 12 months/yr = 6.8 mild events per patient per year. This was conservative in that we multiplied by 1 for the 65 patients reporting 1 to 2 events; if 1.5 is used as the midpoint of 1 to 2 events then there were 8.2 events per year. When the prevalence of patients with any hypoglycemic event was ascertained (mild or severe), there were 105 unique patients (because there were some reports of yes to Q1 and Q2 for the same patient), representing 37% of the sample. This stands in contrast to only 3 of these patients (1% of sample) who had medically coded hypoglycemic events by clinicians’ assessment of hypoglycemia using ICD-9 codes. The difference in hypoglycemia detection was statistically significant using the McNemar test.



The frequency of reported OTC and herbal use among patients with diabetes is in Table 3. Nearly one-quarter of people with diabetes reported taking an OTC medication or an herbal supplement; 4% reported that this was taken “for diabetes” (not in table). None of the data reported herein would be present in any medically coded record or even routinely collected and recorded in the clinical note.



Back to Top | Article Outline


DARTNet was used to pilot test point-of-care data collection on hypoglycemic events among patients with diabetes and ask about OTC medication and herbal supplement use. These domains have implications for comparative effectiveness and also for safety and tolerability. We found hypoglycemia rates and supplement utilization similar to that reported by prospective surveys of other community samples.5,9–11 The use of medically coded events using ICD-9 codes is the traditional means of recording and measuring safety detection, and it is the standard in claims data. This pilot study demonstrated what is already known: patient surveys are highly sensitive but not very specific, in contrast to ICD-9 codes that are highly specific and not very sensitive. Even using enhanced clinical record data such as laboratory tests for liver injury provides additional specific—even definitive—information, yet is not sensitive because not all patients get the tests. This DARTNet point-of-care pilot has shown that sensitive measures can be introduced at the point of care, triggered for specific patients using an electronic query. Using similar techniques, it might be possible to introduce a measure that is both sensitive and specific and use the point of care methods to obtain that data, for example, have patients bring in glucometer readings and report at their visit. Practices could tailor the procedures for obtaining this information, and share it with other practices using the DARTNet learning community. In addition, it would combine the best elements of comparative effectiveness research with elements of the practical clinical trial.12

DARTNet is designed and functioning as an OCER Laboratory. DARTNet was funded to develop and pilot test a cooperative research network for conducting studies on therapeutic safety and effectiveness using electronic health information. The primary goals of the project are to improve public knowledge about health outcomes more quickly than traditional research approaches can. The system eventually should be able to take advantage of the power of networks to increase the ability to detect important sentinel events, identify rare adverse effects, and assess long-term outcomes of treatment. The network spans multiple health care organizations to explore the impact of new innovations that affect small populations (for which no single care delivery system would have a sufficient sample size). This approach enhances the OCER model and provides a location for conducting practical clinical research.

Under a current Agency for Healthcare Research and Quality task order, DARTNet is now scaling up to include a much larger number of clinics, including specialty clinics, and other types of health care data (such as hospital and medication fulfillment data). In this way, DARTNet promises to develop into a virtual integrated clinical and research data system—the US analog to the UK General Practice Research Database (GPRD). DARTNet's design for next-generation comparative effectiveness and safety could also surpass the GPRD in the sense that DARTNet is designed to do some things that the GPRD cannot: conduct point-of-care data collections based on electronic queries and function as a learning community to improve care. The OCER study that will be used to test DARTNet capabilities is on depression treatment, and specifically on the use of second-generation antidepressants for depression treatment of youth and adult patients. Again, we will undertake a point-of-care prospective data collection. We will focus on the greatest missing variable in CER research on depression: severity of illness and its associated measure of treatment effectiveness. To study severity and to obtain subsequent care improvement, we will recruit DARTNet practices to use the PHQ-2 and PHQ-9 for screening and assessment.13 This will be more complex and burdensome than the diabetes survey and has more potential to improve the initiation and monitoring of care.

Regarding the point of care data collection, the 33% response rate was more impressive than the number alone might convey. Features of this pilot study were not designed to maximize response rate and might have even worked against it: data collection only ran for a brief 2 week block at each clinic once it started, and there was little training involved for the clinic staff; it seemed that by the time some of the staff figured out the pattern for obtaining data the pilot was completed. An examination of the tracking database revealed that some days were probably less hectic than other days. When responses were sorted by practice and date, responses were clustered in time; this suggested that staff had more time to distribute and return the surveys on some days or that there were more patients with diabetes seen those days. Although no single practice had a perfect response rate, there was wide variation (13%–64%) that likely reflected variation in practice workflow and organization. There is also the possibility that surveys were completed and did not get faxed to CINA for input into the database. We concluded that the response rate of 33%—possibly considered high by survey standards for this procedure, meaning a one-time, provider-based patient survey with no compensation/incentives or practice staff training—should be capable of improvement under the process subsequently implemented by DARTNet. This also indicates the feasibility of a point-of-care patient survey tool as a viable source of additional information on patient-reported outcomes not generally accessible in the EMR. A more developed plan involving the practice staff in workflow issues and considering ways to allow the patient to complete the form with little to no staff intervention, are all viable options for continued study.

Limitations of this study include some of the traditional limitations associated with observational studies: we observed only the care provided in the practice, and therefore no self pay care in the community, such as consultation or remedies from a commercial herbalist or a natural foods grocer. In addition, surveys were triggered by visits, so patients who did not seek care during the brief data collection windows were not surveyed. However, all patients at all visits during the data collection window who met criteria were eligible to be surveyed. Finally, omitted variable bias will be a persistent source of possible bias but certainly EHR combined with supplemental point-of-care data reduce this threat when compared with clinical data alone. It is unclear in what direction these limitations might bias the samples, but nevertheless they exist. This pilot study focused on the process of EHR-triggered point-of-care data collection, but care should be used to design studies to minimize these measurement threats. We look forward to reporting future use of the DARTNet platform and learning community in service to better community treatment, that leverages the potential of practice-based networks.14

Back to Top | Article Outline


1. Pace WP, Cifuentes M, Valuck RJ, et al. An electronic practice-based network for observational comparative effectiveness research. Ann Intern Med. 2009;151:338–340.
2. Pace W, West D, Valuck R, et al. Distributed Ambulatory Research in Therapeutics Network (DARTNet): Summary Report. (Prepared by University of Colorado DEcIDE Center under Contract No. HHSA 29020050037I-TO2). Rockville, MD: Agency for Healthcare Research and Quality; 2009.
3. Schneeweis S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58:323–337.
4. Schneeweiss S, Patrick AR, Sturmer T, et al. Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomized trial results. Med Care. 2007;45:S131–S142.
5. Leese GP, Wang J, Broomhall J, et al. Frequency of severe hypoglycemia requiring emergency treatment in Type 1 and Type 2 diabetes: a population-based study of health service resource use. Diabetes Care. 2003;26:1176–1180.
6. CINA Quality Suite™. Available at: Accessed June 16, 2009.
7. American Diabetes Association Workgroup on Hypoglycemia. Defining and reporting hypoglycemia in diabetes: a report from the American Diabetes Association Workgroup on Hypoglycemia. Diabetes Care. 2005;28:1245–1249.
8. Crawford C. Effectiveness of cinnamon for lowering hemoglobin A1C in patients with type 2 diabetes: a randomized, controlled trial. J Am Board Fam Med. 2009;22:507–512.
9. Donnelly LA, Morris AD, Frier BM, et al. Frequency and predictors of hypoglycemia in Type 1 and insulin-treated Type 2 diabetes: a population-based study. Diabet Med. 2005;22:749–755.
10. Zammitt NN, Frier BM. Hypoglycemia in Type 2 diabetes: pathophysiology, frequency, and effects of different treatment modalities. Diabetes Care. 2005;28:2048–2061.
11. Barnes PM, Bloom B, Nahin RL. Complementary and alternative medicine use among adults and children: United States, 2007. Natl Health Stat Report. 2008;12:1–23.
12. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290:1624–1632.
13. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Internal Med. 2001;16:606–613.
14. Mold J, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med. 2005;3(Suppl 1):S12–S20.

comparative effectiveness; electronic health record (EHR); diabetes mellitus; oral diabetes medication; hypoglycemia; herbal; OTC; point of care; DARTNet; patient survey

© 2010 Lippincott Williams & Wilkins, Inc.