Clinical-practice guidelines (CPGs) are often developed, disseminated, and implemented as part of the disease management programs of managed care organizations.1 This is done on the assumption that CPGs, by providing practitioners with evidence-based recommendations enhanced by insights from “real-world” clinical practice, effectively improve the quality, efficiency, and consistency of health care and may even reduce costs.2,3 Despite these efforts and expectations, a major body of literature documents the limited effect CPGs generally have on changing physicians’ behavior.4
Shortell and colleagues5 have stated that, to improve quality of health care, evidence-based medicine, which relates to the content of clinical care, must be implemented through evidence-based management, which is the knowledge of how to put content into routine practice. Evidence-based health services management applies this idea by incorporating evidence from well-conducted management research into the decision-making and managerial processes of health services organizations.6 Implementing evidence-based management requires the cultivation of an evidence-based organizational culture that supports and encourages innovation, experimentation, and data collection and analysis, as well as the development of critical appraisal skills among managers.7 In this article, we hope to contribute to efforts to better articulate this notion of organizational culture by reporting on an experience in implementing drug utilization guidelines in a large Israeli managed care organization. This case involved the entire cycle of developing, updating, implementing, and evaluating the outcomes of CPGs, with all of these stages carried out in a conscious attempt to achieve a fit with the organizational environment.
Israel’s National Health Insurance Law mandates that the country’s four competing health funds provide a standard basket of health services to their members, who may move between the funds twice annually. Financed by capitation payments adjusted for age, the plans face the challenge of maximizing health outcomes and member satisfaction while subject to strict budget constraints. Drug utilization may be the area in which this set of constraints is most salient.
Drug utilization analyses conducted in the Leumit Health Fund (LHF) of Israel, a national managed care organization that provides health coverage to more than 700,000 members (∼11% of the population), have consistently demonstrated low adherence rates among primary care physicians to the LHF’s recommendations for choice of antibiotic for empiric treatment (without urine culture) of treating acute, nonrecurrent, uncomplicated cystitis in adult women.8,9 These findings are of particular significance in view of the rising levels in Israel of uropathogenic E. coli resistance to antibiotics, particularly fluroquinolones.10–12 Because the prescription of inappropriate antibiotics increases the prevalence of resistant organisms, efforts must be made to encourage physicians to prescribe these drugs as precisely as possible.13–16 This has led LHF’s management to recognize the need to develop new strategies for disseminating and implementing CPGs if they are to get more MDs to adhere to the next, revised round of guidelines. The purpose of this study was to design and implement a program to foster the development of an evidence-based culture in which CPGs can be formulated, promulgated, and effectively implemented.
Method for Implementing a Process of Change: A Four-Stage Model
The test case for this study was the development, dissemination, and implementation of CPGs for the treatment of uncomplicated cystitis in adult women. This program was conducted in four stages. These stages are presented in the next sections following a narrative description intended to capture the iterative nature of interventions and feedback within the organizational context in a manner appropriate to a focus on organizational culture.
Stage I: Develop tools and methods to evaluate adherence rates
The tools and methods developed for analyzing physicians’ prescribing behavior for the treatment of uncomplicated cystitis in adult women in the LHF have been reported at length previously.8,9 We derived the data on physicians’ prescribing behavior from LHF’s electronic patient record (EPR) system. Israeli health plans all have developed and maintain EPRs that enable monitoring and evaluation of processes of care. A relational database module directly linked to EPR data was created to facilitate drug utilization studies for the analysis of prescribing behavior for patients diagnosed with specific diseases (ICD-9 codes registered during visit). Once this module was created, we were able to monitor indication or pharmacological category-specific prescribing practices independently, thus eliminating the need to request programming resources from LHF’s information technology division for subsequent analyses. This facilitated the inception of a managerial routine of national longitudinal surveillance of prescribing patterns conducted by LHF’s medical division. Additionally, this tool enabled the medical division to generate data to enhance managerial processes such as drug policy formulation and measurement of effects of policy implementation on drug utilization.17
Stage II: Update guidelines
CPGs need to present current scientific knowledge to be useful to practicing physicians.18 Changes in evidence on the benefits of interventions, evidence supporting the effectiveness of currently recommended practices, and the array of resources available to health care providers often necessitate updating of guidelines.19 Similarly, guidelines may be updated when evidence from the initial version of a CPG has been either supplemented or improved.20
After we had completed Stage I of this project, changes necessitated a revision of LHF’s antibiotic policy. The first version of the guideline recommended trimethoprim plus sulfamethoxazole (TMP+SMX) as the drug combination of choice for treating uncomplicated cystitis in adult women. This policy required revision to account for uropathogenic E. coli resistance rates to TMP+SMX that already exceeded 20%.21 Additionally, the introduction into the market of generic equivalents of such oral antibiotics as ofloxacin, ciprofloxacin, cefuroxime, and nitrofurantoin begged a revised cost-minimization analysis of treatment modalities for common infectious diseases, including uncomplicated cystitis.
The guideline was updated within the framework of a workshop for family practice residents.22 Extensive literature searches, reviews of guidelines from other health systems (both Israeli and international), expert opinions, and information from Stage I’s drug utilization analyses all informed the updating process. We also used data generated by LHF’s central laboratory to calculate current rates of uropathogenic E. coli in its patient population for the antibiotics commonly used to empirically treat uncomplicated cystitis in adult women.10 In particular, we learned that, because of the sporadic supply of difficult-to-register nitrofurantoin, about 50% of the cases were treated with fluroquinolones, particularly ofloxacin, despite it not being highly recommended.23 We also learned that fewer than 5% of the ofloxacin treatments stopped after the recommended duration of three days; most lasted 5 to 10 days, thus increasing the risk of uropathogen resistance, while also wasting both drugs and money.24,25
From this information we ascertained that, although nitrofurantoin was chosen as the drug of choice in the updated guideline, careful consideration had to be given to a second-line therapy, because nitrofurantoin may periodically be unavailable. Additionally, we recognized that the evidence-based duration of treatment of three days for ofloxacin, the drug recommended for second-line therapy, had to be emphasized more strongly than in the previous version.
Stage III: Choose and design technology for disseminating and implementing guideline
As indicated above, a major goal of this study was to put flesh on the notion of evidence-based culture, particularly within the LHF. Given this focus on culture, we strove, when developing a method for disseminating and implementing guidelines, to use both qualitative and quantitative methodologies to obtain ideas from the target population, MDs practicing in LHF clinics. Our primary research objective was to gain insights from practicing physicians into nonadherence to guidelines, why past efforts to promulgate CPGs had failed, and what tools and methods might be more successful. Initially, we extensively searched the literature for interventions that had been effective in other organizations and used our findings to generate hypotheses as to what might best improve LHF physicians’ adherence to guidelines for choice of drugs, particularly antibiotics. We postulated that the observed lack of adherence was due both to deficient dissemination techniques, which were responsible for MDs’ lack of awareness of the content of the guidelines, and to an absence of implementation mechanisms that could modify physicians’ prescribing practices. On the basis of our agreement that the prerequisites for changing behavior are awareness, understanding, and acceptance of both the problem and its solution,26 we hypothesized that a multipronged strategy composed of both an educational and an administrative intervention would improve MDs’ adherence to a CPG on prescribing antibiotics for uncomplicated cystitis.
Stage IIIa: Conduct focus groups with target population physicians.
In an evidence-based culture, physicians’ input is an indispensable component in designing interventions for the dissemination and implementation of CPGs. We developed a methodology that, by presenting the prescribing data accrued in Stage I and the antibiotic resistance patterns calculated in Stage II to small groups of physicians, elicited group discussions that furthered our understanding of the factors underlying MDs’ lack of adherence to CPGs. Also, knowing that the prerequisites for changing behavior are awareness, understanding, and acceptance of both the problem and its solution, we strove in those group discussions to ascertain what messages might successfully instill in the target population of physicians the awareness and understanding of the problem of antibiotic resistance necessary to change their prescribing behavior.
We used purposive sampling to recruit focus group participants.27 To make the focus groups representative, we invited all primary care MDs practicing in LHF clinics in three regions of Israel. The regions were carefully selected to provide representation of all major demographic subpopulations (including ethnic, religious, new immigrants, rural, and urban) of both MDs and patients. These sessions, which comprised 6, 10, and 15 physicians, were conducted in 2006 during regular working hours or during the afternoon break immediately after clinics closed to minimize barriers to participation and to avoid possible self-selection of MDs. Electronic voting technology was used to stimulate discussion. We presented the results of pharmacoepidemiology studies that reported the current patterns for LHF’s patient and MD populations. Immediate feedback concerning the findings then generated group discussion, conducted with nominal group technique.28 We asked the participants what they believed were the barriers to implementing guidelines and what tools and methodologies might be used to successfully promulgate guidelines in the future.
Stage IIIb: Survey physicians.
We then used the major themes ascertained from the focus groups (Table 1) to formulate a questionnaire to survey a large sample of LHF’s primary care physicians. The primary objective of the survey, conducted in 2007, was to quantitatively rank potential behavior-changing strategies. The questionnaire also addressed personalized feedback, a theme discussed at length in the focus groups, asking the physicians who they thought would be the most appropriate authority to convey feedback and what the content of the feedback should be.
The survey participants rated (on a five-point Likert scale; 1 = completely disagree; 2 = slightly agree; 3 = somewhat agree; 4 = agree; 5 = very much agree) their agreement with statements concerning attitudes toward CPGs, methods for promulgating CPGs, types of content that should be included in a prescription audit, and the appropriateness of a variety of people for providing such feedback. Two multiple-choice questions ascertained whether the physicians had received the LHF’s booklet containing the guidelines for treating infectious diseases commonly encountered in the community setting and how, if at all, it was brought to their attention that the guidelines appear on the LHF’s intranet site.
We validated the questionnaire through three pretests with LHF physicians, whose debriefing suggestions for improving the questionnaire were recorded. We uploaded the final, validated questionnaire (Cronbach alpha = 1.0) to an in-house survey program on the LHF’s intranet site. All 319 primary care physicians practicing in LHF clinics were electronically notified of the survey when they logged in to access their patients’ records. Double-clicking on a link at the end of the notification document opened the site with the survey questionnaire. At the end of each week, the LHF’s information technology division sent us an updated list of physicians who had not yet responded to the survey. We then asked the LHF’s district physicians (physician–managers who supervise MDs in their district) to contact the nonresponders and encourage them to complete the survey. The survey was conducted for five weeks, after which we compiled and recorded the downloaded data onto a Microsoft Excel spreadsheet. We conducted a statistical analysis using SPSS version 15. For the Likert scale questions, we calculated average scores and 95% confidence intervals. We also calculated distribution of answers (%) for the two multiple-choice questions with a chi-square test performed for statistical significance.
Of the 319 eligible physicians, 226 (70.8%) participated in the survey. Demographic information on both respondents and nonrespondents appears in Table 2. We found no statistically significant differences in MD characteristics between the two subpopulations.
The survey questions and results appear in Table 3. The physicians ranked highest those themes that emphasize the scientific and professional aspects of pharmacotherapy, while minimizing the importance of its administrative and economic facets. This attitude is best illustrated by the high rank given to medical experts both as sources of feedback and as the integral element of professional lectures, the preferred method for promulgating guidelines. Conversely, the physicians expressed low regard toward nonphysicians as auditors regardless of their position in the organization’s administrative hierarchy. (Interestingly, this finding also allayed our concerns that lack of anonymity may have biased the study, seeing as these physicians felt free to register low ranks to their immediate superiors.) The rank given to a pharmacist as an auditor, superior to non-MDs but inferior to administrative MDs, strengthened our analysis by delineating a trend of escalating agreement as the perceived professional level of the authority increases. Regarding content of feedback, the physicians judged the scientific foundation for the CPGs, as represented by data on resistance patterns, to be a potentially convincing message; they found cost data less so.
After analyzing the survey results, we compared the physicians’ suggestions and rankings of interventions with the published literature on CPG dissemination and implementation techniques. Strategies recommended by the LHF physicians, such as educational interventions29 and auditing and feedback,30–35 have shown promise in improving prescribing behavior. Although computer interventions and academic detailing1,35–37 have also been shown to be effective, a feasibility study by the LHF found these options unviable because of a lack of resources.
Systematic reviews have generally recommended the implementation of multipronged interventions,30,36,38 such as feedback combined with educational interventions,39 particularly after addressing local barriers to change.40 Although Steinman et al41 found that multipronged interventions composed of audit and feedback were less effective than education alone, the authors claim that confounding may partially account for this finding. On the basis of these findings, we chose to implement a multipronged design composed of an educational intervention and individualized prescriber feedback. We hypothesized that this was necessary to overcome barriers to both effective dissemination and implementation.
Stage IV: Conduct randomized controlled trial of interventions
Shortly after completing the data analysis, we became aware of an upcoming two-day conference for family physicians. These weekend (Friday–Saturday) conferences, which are recognized for CME credits, take place four times per year, and all of LHF’s salaried MDs are invited on a rotating basis to attend (they pay a symbolic fee and may bring their spouses). Invitations rotate according to regional quotas to ensure that no region is left without medical care during these two days (clinics are open for a half a day on Fridays). We viewed this conference as a twofold opportunity. First, although the physicians had ranked a lecture by an expert at a conference as their preferred intervention method, we lacked the funding for such an endeavor. Second, because these conferences are routine annual events that all MDs are invited to attend, we decided that initiating a routine of measuring possible changes in MD behavior after the conference fit well into our research agenda of constructing an evidence-based culture in which all processes in the organization should be subject to quantitative scrutiny and evaluation. Accordingly, we requested that a 10-minute slot be added to the program for a review of the guidelines we wished to promulgate. This became the first stage of our two-stage multipronged intervention program.
The content of the lecture included the information recommended by the physicians in Stage II: current uropathogenic E. coli antibiotic resistance rates (the scientific rational behind the recommendations), current population-based prescribing patterns, and a brief review of the guideline, with an emphasis on the duration of treatment with ofloxacin when nitrofurantoin, the drug of choice, was unavailable.
Approximately five months later, we took the group of MDs who had attended the lecture and another group who had not, and randomized each group into two subgroups. We then sent a letter containing personalized feedback on prescribing practices to the MDs in two of the subgroups, thereby subdividing the study population into four cohorts: physicians exposed to both interventions (n = 32), physicians only receiving feedback via personalized letter (n = 130), physicians who only attended the lecture (n = 29), and a control group who were not exposed to either intervention (n = 107).
The letter, designed according to the major findings of the survey, contained notification that nitrofurantoin was now the drug of choice and that TMP+SMX was no longer recommended and emphasized that, should nitrofurantoin not be available, ofloxacin should be prescribed—but only for three days. The letter also included a table with current uropathogenic E. coli antibiotic resistance rates and an individualized report for each MD showing the distribution of antibiotics he or she had prescribed for uncomplicated cystitis in the previous six months, including his or her rate of adherence to the guideline for duration of treatment with ofloxacin. Shortly after the lecture at the conference, nitrofurantoin was again unavailable in Israel. We therefore waited until a steady supply was renewed before sending the letter, which included notification that the drug was currently in stock. In view of the findings of the survey, the letter was signed by a professor of family medicine and the LHF’s infectious disease consultant. Copies were not sent to anyone besides the physician, particularly not to managers; we wanted the intervention to be remedial and educational, not punitive or intimidating.
The characteristics of the MDs in each subgroup are presented in Table 4. The primary outcome variables analyzed were the compound variable, the rate of eligible cases treated with either nitrofurantoin or second-line therapy of ofloxacin for three days, and each of these variables individually. We used the methods developed in Stage I of this study to evaluate the prescribing behavior of the three intervention groups, before and after the interventions, as compared with the control group. SPSS version 15 was used to conduct statistical analysis. The tools and methods developed in Stage I were used to measure prescribing practices. Rates of adherence and rate differences between the 12-month preintervention period and the 5-month post-lecture-intervention and 4-month post-letter-intervention period were calculated with 95% confidence intervals. On the basis of the findings of Steinman et al,41 we expected to achieve a rate of improvement of 10.6% to 13.9%.
The results of the trial are presented in Table 5. We observed an increased rate of adherence to the guidelines of about 7% among physicians who attended the lecture in the five months after that intervention (the period between the lecture and the letter) compared with a decrease of 4.2% to 5.7% in the controls. Adherence to the recommendations for duration of treatment with ofloxacin improved by 13.1% to 17.1% among physicians in the intervention groups. No significant change for this variable was observed in the control groups. During this period, when supplies of nitrofurantoin were diminished, the proportion of cases treated with that drug of choice remained constant (no significant change) in the intervention groups while decreasing in the control groups by 4.4% to 5.3%.
During the four months after the letter intervention, adherence to the guidelines improved by 19.4% (95% CI = 20.6, 26.7) in the letter-only group and by 18.3% (95% CI = 11.6, 24.9) in the multipronged group. No significant change was observed in the lecture-only group (4.9%, 95% CI = −0.7, 10.5) or the control group (1.2%, 95% CI = −1.4, 3.8).
In this study, we succeeded in developing an end-user-sensitive, data-driven process for formulating, disseminating, and implementing CPGs tailored to an HMO’s target MD population. The tools and data-generating capabilities designed and constructed in Stage I were integral components of all subsequent stages of the program. In Stage II we expanded the scope of the utility of these data by implementing a hybrid methodology composed of qualitative and quantitative analyses based on prescribing data. Through this modality, we achieved our objective of developing a technique that presents population-based drug utilization data to practicing physicians and elicits their interpretations of those findings. The electronic voting technology proved to be a highly effective icebreaker for initiating group discussion. Because people in general, and physicians in particular, are naturally apprehensive about exposing their ignorance before their peers, the anonymity of this modality facilitated the nonthreatening, nonjudgmental environment essential for promoting the requisite self-disclosure for evaluating physicians’ attitudes and knowledge levels. This factor may be particularly relevant during sessions in which the researcher wishes to elucidate knowledge gaps and promote recognition of misconceptions concerning current clinical practice. Likewise, groups often include relatively docile participants whose input may be hindered or stifled by the “overparticipation” of overbearing or senior participants who dominate the conversation. Accordingly, this modality, which combines two techniques that ensure that all participants have an opportunity to contribute to the process, may be particularly appropriate for conducting small-group discussions in highly hierarchical environments such as the clinical setting.
The letter intervention significantly influenced physicians’ prescribing patterns, improving both their choice of drug and their adherence to the recommendation that ofloxacin treatment last only three days. Although the lecture intervention seemed effective in the short run, the lecture-only group regressed during the third period (post letter), indicating that the effect of this technique does not last unless reinforced, as seen in the group exposed to the multipronged intervention. Our results do not indicate any advantage of a multipronged intervention composed of education and individual feedback over an intervention that uses only individual feedback. An explanation for this finding may be that the letter was both effective enough to surmount barriers to efficient dissemination and convincing enough to overcome the barriers impeding implementation. Specifically, the mechanism of a personalized letter may have been effective in making the physicians aware of the guidelines while the content, notification of nitrofurantoin’s availability, brief description of the scientific rationale behind the recommendations, and individualized auditing of prescribing behavior were successful in bringing MDs to adhere to these recommendations.
It should be emphasized that behavioral changes of this magnitude are greater than others described in the literature. We suggest that this outcome is due to the fact that the guideline was developed, implemented, and evaluated in a manner emphasizing its fit to the organizational culture and sensitive to input from the key stakeholder group: physicians. Implementation of mixed methodologies, which included qualitative data accrual via focus groups and then a quantitative survey of the target MD population, can therefore be viewed as keys to the success of this process.
By evaluating the chosen interventions in a randomized controlled trial, we succeeded in introducing into the managerial decision-making process of the LHF the element of experimentation necessary for nurturing an evidence-based culture. Although this study focused exclusively on improving the prescribing precision of antibiotics, with careful thought it may have implications regarding prescription of other classes of drugs, or even clinical practices other than drug therapy. Whereas all of the commonly used antibiotics for treatment of uncomplicated cystitis are currently off-patent, this study was not designed to counteract the marketing efforts of drug companies to promote expensive products not highly recommended by the organization. Accordingly, the interventions described were designed to communicate a strong, universally accepted message of the association between suboptimal prescribing of antibiotics and uropathogen resistance and were devoid of any economic content. This model may therefore be useful for promoting other pertinent, evidence-based messages via a mechanism for presenting individual data while being sensitive to professional respect. These may include preventing prescription of drugs for patients for whom well-established contraindications may exist, improving adherence to recommendations for therapeutic drug monitoring, INR testing for warfarinized patients, or HbA1c testing in diabetics. Additionally, with the growing number of drugs entering the generic market, this study may be applicable for promoting evidence-based prescription of other classes of drugs where the incentive for change is no longer economic, but clinical. Because the effectiveness of such efforts will almost always depend on the degree that physicians are convinced of the scientific merit of these recommendations, our findings indicate that partnerships with academic, nonmanagerial medical educators should be routine practice for enhancing these processes.
This project demonstrated that evidence based management can be put into practice for improving prescribing precision of antibiotics in an HMO. This process, which included both quantitative and qualitative methodologies culminating in a randomized controlled trial of a tailored intervention, may be feasible in other health care systems. This study provides a template for introducing the component of experimentation essential for cultivating an evidence-based culture. By conducting a methodologically robust study, we created a template for future analyses that will enable our organization to evaluate new ideas among this defined population of physicians and patients. Additionally, by creating a model that relies on integrating extant resources and data into organizational change processes, this study may provide a model for development and evaluation of interventions tailored to the needs of other health care systems.
The authors wish to express their gratitude to the Information Technology division of the Leumit Health Fund for their invaluable assistance in conducting the intranet-based survey.
This study was partially funded through a research grant from The Israel National Institute for Health Policy and Health Services Research and through an educational grant from Schering Plough Israel.
Portions of this study were presented at the 23rd International Conference on Pharmacoepidemiology and Therapeutic Risk Management, August 19–27, 2007, Quebec City, Canada, and at the International Society of Pharmacoeconomics and Outcomes Research 14th International Meeting, May 16–20, 2009, Orlando, Florida. This study is part of Dr. Kahan’s PhD dissertation.
1 Armstrong P. Clinical and economic outcomes of an ambulatory urinary tract infection disease management program. Am J Manag Care. 2001;7:269–280.
2 Grol R. Successes and failure in the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39: II-46–II-54.
3 Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Potential benefits, limitations and harms of clinical guidelines. BMJ. 1999;318:527–530.
4 Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follw clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458–1465.
5 Shortell SM, Rundall TG, Hsu J. Improving patient care by linking evidence-based medicine and evidence-based management. JAMA. 2007;298:673–676.
6 Kovner AR, Rundall TG. Evidence-based management reconsidered. Front Health Serv Manage. 2006;22:3–22.
7 Walsh K, Rundall TG. Evidence-based management: From theory to practice in health care. Milbank Q. 2001;79:429–457.
8 Kahan E, Kahan NR, Chinitz DP. Urinary tract infection in women: Physician’s preferences for treatment and adherence to guidelines: A national drug utilization study in a managed care setting. Eur J Clin Pharmacol. 2003;59:663–668.
9 Kahan NR, Chinitz DP, Kahan E. Longer than recommended empiric antibiotic treatment of urinary tract infection in women: An avoidable waste of money. J Clin Pharm Ther. 2004;29:59–63.
10 Kahan NR, Chinitz DP, Waitman DA, Dushnitzky D, Kahan E, Shapiro M. Empiric treatment of uncomplicated UTI with fluoroquinolones in older women in Israel. Another lost treatment option? Ann Pharmacother. 2006;40:2223–2227.
11 Nesher L, Novack V, Riesenberg K, Schlaeffer F. Regional community-acquired urinary tract infections in Israel: Diagnosis, pathogens, and antibiotic guidelines adherence: A prospective study. Int J Infect Dis. 2007;11:245–250.
12 Rock W, Colodner R, Chazan B, Elias M, Raz R. Ten years surveillance of antimicrobial susceptibility of community-acquired Escherichia coli and other uropathogens in northern Israel (1995–2005). Isr Med Assoc J. 2007;9:803–805.
13 Avorn J, Solomon DH. Cultural and economic factors that (mis)shape antibiotic use: The nonpharmacologic base of therapeutics. Ann Intern Med. 2000;133: 128–135.
14 Hooton TM, Levy SB. Antimicrobial resitance: A plan of action for community practice. Am Fam Physician. 2001;63: 1087–1096.
15 Metlay JP, Shea JA, Crossette LB, Asch DA. Tensions in antibiotic prescribing. Pitting social concerns against the interests of individual patients. J Gen Intern Med. 2002;17:87–94.
16 Donnan PT, Wei L, Steinke DT, et al. Presence of bacteriuria caused by trimethoprim resistant bacteria in patients prescribed antibiotics: Multilevel model with practice and individual patient data. BMJ. 2004;328:1297.
17 Kahan NR, Chinitz DP, Waitman DA, Kahan E. When gatekeepers meet the sentinel: The impact of a prior authorization requirement for cefuroxime on prescribing behaviour of community based physicians. Br J Clin Pharmacol. 2006;61:341–344.
18 Shekelle PG, Rhodes S, Ortiz E, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: How quickly do guidelines become outdated? JAMA. 2001;286:1461–1467.
19 Shekelle P, Ecles MP, Grimshaw JM, Woolf SH. When should clinical guidelines be updated? BMJ. 2001;323:155–157.
20 Eccles M, Rousseau N, Freemantle N. Updating evidence-based clinical guidelines. J Health Serv Res Policy. 2002;7:98–103.
21 Warren JW, Abrutyn E, Hebel JR, Schaeffer AJ, Stamm WE. Guidelines for antimicrobial treatment of uncomplicated acute bacterial cystitis and acute pyelonephritis in women. Clin Infect Dis. 1999;29:745–758.
22 Kahan NR, Fogelman Y, Waitman DA, et al. Teaching evidence-based medicine in a managed care setting: From didactic exercise to pharmacopolicy development tool. Am J Manag Care. 2005;11:570–572.
23 Kahan NR, Kahan E, Waitman A, Chinitz DP. An evaluation of the potential economic impact of implementing an updated guideline for choice of antibiotic for the empiric treatment of uncomplicated UTI in women: A national pharmacoeconomic study in a managed care setting. Isr Med Assoc J. 2004;6:588–591.
24 Kahan NR, Chinitz DP, Kahan E. Physician adherence to recommendations for duration of empiric antibiotic treatment for uncomplicated urinary tract infection in women: A national drug utilization analysis. Pharmacoepidemiol Drug Saf. 2004;13: 239–242.
25 Kahan NR, Chinitz DP, Kahan E. Longer than recommended empiric antibiotic treatment of urinary tract infection in women: An avoidable waste of money. J Clin Pharm Ther. 2004;29:59–63.
26 Grol R. Implementing guidelines in general practice care. Qual Health Care. 1992;1: 184–191.
27 Shkedi A. Words of Meaning. Qualitative Research—Theory and Practice. Tel-Aviv, Israel: Ramot Tel-Aviv University Press; 2003.
28 Glasper A, Richardson J, Whiting M. The highs and lows of learning to be a children’s nurse. Paediatr Nurs. 2006;18:22–26.
29 Lundborg CS, Wahlstrom R, Oke T, Tomson G, Diwan VK. Influencing prescribing for urinary tract infection and asthma in primary care in Sweden: A randomized controlled trial of an interactive educational intervention. J Clin Epidemiol. 1999;52:801–812.
30 Gross PA, Duressa P. Implementing practice guidelines for appropriate antimicrobial usage. A systematic review. Med Care. 2001;39:II-55–II-69.
31 Lagerlov P, Loeb M, Andrew M, Hjortdahl P. Improving prescribing behaviour through reflection on guidelines and prescription feedback: A randomized controlled trial. Qual Health Care. 2000;9:159–165.
32 Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care. 2006;15:433–436.
33 Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit and feedback: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006;2:CD000259.
34 Zwar N, Wolk J, Gordon J, Sanson-Fisher R, Kehoe L. Influencing antibiotic prescribing in general practice: A trial of prescriber feedback and management guidelines. Fam Pract. 1999;16:495–500.
35 Pearson SA, Ross-Degna D, Payson A, Soumerai SB. Changing medication use in managed care: A critical review of the available evidence. Am J Manag Care. 2003;9:715–731.
36 Gross PA, Greenfield S, Cretin S, et al. Optimal methods for guideline implementation. Conclusions from Leeds Castle meeting. Med Care. 2001;8: II-85–II-92.
37 Lu CY, Ross-Degnan D, Soumerai SB, Pearson SA. Interventions designed to improve the quality and efficiency of medication use in managed care: A critical review of the literature—2001–2007. BMC Health Serv Res. 2008;8:75.
38 Shojania KG, Grimshaw JM. Evidence-based quality improvement: The state of the science. Health Aff (Millwood). 2005;24:138–150.
39 Madridejos-Mora R, Amado-Guirado E, Perez-Rodriguez MT. Effectiveness of the combination of feedback and educational recommendations for improving drug prescription in general practice. Med Care. 2004;42:643–648.
40 Arnold SR, Strauss SE. Interventions to improve antibiotic prescribing in ambulatory care. Cochrane Database Syst Rev. 2005;4:CD003539.
41 Steinman MA, Ranji SR, Shojania KG, Gonzales R. Improving antibiotic selection. A systematic review and quantitative analysis of quality improvement strategies. Med Care. 2006;44:617–628.