Secondary Logo

Journal Logo

Special Theme: Complementary, Alternative, and Integrative Medicine: SPECIAL THEME RESEARCH REPORTS

Randomized Trial of an Internet Curriculum on Herbs and Other Dietary Supplements for Health Care Professionals

Kemper, Kathi J., MD, MPH; Amata-Kynvi, Andey, RN; Sanghavi, Darshak, MD; Whelan, Julia S., MS; Dvorkin, Lana, PharmD; Woolf, Alan, MD, MPH; Samuels, Ronald C., MD, MPH; Hibberd, Patricia, MD

Author Information
  • Free


Herbs and other dietary supplements are the most commonly used complementary and alternative medical (CAM) therapies in the United States.1,2,3,4,5 Up to 50% of patients with serious or chronic health conditions use herbs and other dietary supplements, frequently in conjunction with prescription and over-the-counter medications.6,7,8,9 Many persons also rely on herbal remedies as part of their cultural traditions of home therapies for common ailments.10,11,12 However, fewer than half of patients who use CAM discuss their use with their primary physicians.13,14,15

Because of the frequency of use and the potential risks posed by herbs and other dietary supplements, clinicians have been urged to initiate routine discussions about them with patients.16,17 Systematic discussions may uncover direct toxicity, products contaminated with misidentified herbs, heavy metals, and pharmaceuticals, variability in dosing, and adverse interactions between herbs and other dietary supplements and medications.18,19

However, clinicans may lack knowledge about the risks of using these products and the resources to answer patients' questions about them. Few studies have evaluated health care professionals' knowledge, attitudes, or practices regarding herbs and dietary supplements, and educational research has not addressed the most effective ways of educating traditionally trained, practicing clinicians about them.20

As part of our ongoing educational efforts through the Longwood Herbal Task Force (〈〉) and the Center for Pediatric Integrative Medical Education at Children's Hospital in Boston (〈〉), we developed an Internet-based (e-mail) curriculum to educate geographically and professionally diverse health care professionals about herbs and other dietary supplements. We performed a randomized controlled crossover trial to evaluate the feasibility of providing the curriculum and its impact on clinicians' knowledge, attitudes, and clinical practices regarding herbs and other dietary supplements.


In this trial of in-training and practicing physicians, pharmacists, advance practice nurses, and dietitians, potential participants completed a baseline questionnaire in September 2000. Participants were then randomized into immediate intervention versus waiting-list groups. The immediate group received the curriculum over approximately ten weeks, and both groups completed the first follow-up questionnaire within two weeks of the immediate intervention group's last module (December 2000). There was a five-week break for the December and New Year's holidays. The waiting-list group then received the curriculum over approximately ten weeks. Both groups received the second follow-up questionnaire, which was completed by the end of April 2001.

Because it was initially conceived as a feasibility study, we targeted institutions in the local Longwood medical area surrounding Harvard Medical School: the Boston Children's Hospital, Dana Farber Cancer Institute, Harvard Medical School, the Massachusetts College of Pharmacy and Health Sciences, and residents at the Boston Combined Pediatric Residency Program. We also sent invitations to pharmacy directors at Brigham and Women's Hospital, Massachusetts General Hospital, and the Veteran's Administration Hospital in Boston. Altogether, we sent approximately 1,900 invitations via e-mail to advanced practice nurses (∼70), pharmacists (∼25), physician—faculty and medical students (∼1,800), and chief residents (∼four). Because we were unable to obtain complete distribution lists for several departments, in some cases we asked department chairs, division chiefs, and chief residents to forward the invitation to those in their departments or divisions who might be interested. Six weeks after the initial direct e-mail invitation, we repeated the invitation to non-respondents.

The e-mail invitations were widely forwarded. For example, within one week of the initial e-mail invitation, we received responses from physicians in Germany and Australia and from several American dietitians asking whether they could participate. Because the project was initially designed to test the feasibility of delivering curricula via e-mail to diverse health care professionals, we decided to allow a broad spectrum of participants provided we could achieve a large enough sample of each type of professional (arbitrarily set at 30 or more individuals from each profession). With our permission, one registered dietitian forwarded information about the project to two e-mail—based discussion groups of registered dietitians, and those who met the eligibility criteria (described below) were allowed to enroll.


Health care professionals were eligible for the project if they had access to the Internet and e-mail, planned to keep their current e-mail addresses for at least 12 months, and checked their e-mail at least twice weekly. They were excluded if they said that they would not complete both the baseline and the outcome questionnaires or were unwilling to be randomized to wait up to 30 weeks to receive the free curriculum.

Although our anticipated sample size was 200 subjects, 537 persons met inclusion criteria and completed the baseline questionnaire. Most (84%) were in practice or on faculty; the other 16% were students, residents, fellows, or postdoctoral trainees (36 medical students, residents, or fellows, 24 dietitians, 24 pharmacy students, and two nurses). When practitioners and trainees were combined, there were 111 physicians (MD), 46 pharmacists (PharmD), 30 advanced practice nurses (APN), and 350 registered dietitians (RD).

Study Variables

The baseline questionnaire asked questions about demographics (age, gender, race or ethnicity), and knowledge, attitudes, and practices regarding herbs and dietary supplements. It took about 20 minutes to complete over the Internet. The knowledge, attitudes, and practices questions were developed by a multidisciplinary group of clinicians participating in the Longwood Herbal Task Force and were pilot tested at Children's Hospital. The questions were refined, reduced in number, and edited by a staff expert in survey development. The participants had implicit permission to use any available reference materials and were given up to two weeks to complete the questions. The knowledge section of the baseline questionnaire contained ten multiple-choice questions and ten true-or-false questions; Cronbach's alpha test of internal consistency for these 20 questions was .71. The knowledge section was scored as the percentage of correct answers for the 20 questions. The questions about participants' attitudes about herbs and dietary supplements were all in Likert-type format with five potential responses ranging from “strongly agree” to “neutral/not sure” to “strongly disagree.” A confidence score was generated from the ten attitude items that specifically related to participants' confidence in dealing with questions about herbs and dietary supplements; each item was scored as confident (typically strongly agree or agree) or not confident (typically neutral, disagree or strongly disagree, except for two items with reverse wording). The maximum confidence score was 10; a Cronbach's alpha test of internal consistency for these ten questions was .83. The questionnaire also asked questions about participants' practices related to herbs and dietary supplements. We scored participants' communication practices as the sum of responses to four items such as having discussed herbs and dietary supplements with patients and having discussed them with a colleague in the past 30 days. The communication scores ranged from 0 to 4; Cronbach's alpha for this measure was .48. The same questions were asked in the baseline and two follow-up surveys.

Internet Curriculum

The curriculum consisted of three elements. The first was a set of 20 self-instructional modules delivered by e-mail; each module contained a one-to-two sentence case, a single multiple-choice or true-or-false question, and the answer (see the Appendix for an example of a module). The modules were developed by physicians, pharmacists, advanced practice nurses, and dietitians who were members of the Longwood Herbal Task Force. The modules underwent extensive pilot testing and revision prior to use. Each module required less than three minutes to complete and could either be viewed as e-mail text or be completed interactively on the project's Internet site. Two to three modules were delivered weekly by e-mail over ten weeks. The educational modules provided the answers to all knowledge questions on the baseline and follow-up surveys.

The second element of the curriculum was linkage to evidence-based Internet resources about herbs and dietary supplements. Two to four links were provided at the end of each module. For many modules the primary links were to the Longwood Herbal Task Force (〈>); other links included government sites such as the National Institutes of Health Office of Dietary Supplements and the National Center for Complementary and Alternative Medical Therapies, non-profit health groups such as the American Cancer Society and American Heart Association, academic sites such as the University of Texas and Columbia University, and commercial health information sites such as WebMD. Viewing links was left to the discretion of each participant and was not required to complete any module.

The third element of the curriculum was a moderated listserv, an electronic discussion group in which each group was enrolled during its curricular period. Participants were encouraged to post clinical questions to the list, and to answer other participants' questions; they could also choose not to post, but simply to “lurk” and read the postings. Messages were posted two to three times weekly. Participants were allowed to un-enroll for any reason.

Statistical Analyses

The initial sample-size goal of 200 participants was based on previous studies suggesting an improvement in knowledge scores from 67% at baseline to 80% following the curriculum.20 This sample size was based on an estimated a standard deviation of 11%, a dropout rate of 20%, and a power of 90% to detect a difference between groups using a conservative intention-to-treat type of analysis and a two-tailed test with p < .05 considered a significant difference. Based on typical responses to direct-mail solicitations, we anticipated a response rate of 1–1.5% from the 1,900 persons who were invited to participate. Because of the additional group of participants (the RDs), the sample size expanded beyond original expectations.

We assessed the comparability of the study's participants in the two groups at baseline using the chi-squared test or Fisher's exact test for categorical variables and the t-test or Wilcoxon Mann—Whitney test for continuous variables. We compared the changes in knowledge, confidence, and communication, from baseline and both follow-up surveys, using either a t-test or the Wilcoxon Mann—Whitney test as appropriate for parametric and nonparametric data. As described above, we analyzed data using the intent-to-treat principle, with the missing scores (non-responders) conservatively being entered as the average score of the other group. For example, if someone in the immediate intervention group did not complete the knowledge questions for the first follow-up questionnaire, he or she was assigned the average knowledge score of the waiting-list group. This analytic approach tends to favor the null hypothesis.

All participants' responses to the questionnaire were entered directly into a database. For the 5–7% of participants who had technical difficulties with the Internet site and sent responses by fax, the data were entered manually. All data were checked, cleaned, and exported into a standard statistical software package for data analysis.


The immediate intervention and waiting-list groups had similar demographic, professional practice, previous training in herbs and dietary supplements, and geographic location characteristics (see Table 1). The vast majority of participants were women (86%) and Caucasian (88%). The largest minority group was Asian/Pacific Islander (7%), and few participants identified themselves as being Hispanic (2%). Only 127 (24%) of the respondents were from the original target institutions in Boston; the remainder were from a variety of health care settings across the United States and Canada. Most (66%) reported having received some professional education or training about herbs and dietary supplements in the 12 months before the study, and many (75%) reported having read articles in professional journals about the benefits or risks of herbs and dietary supplements within 30 days prior to the study. Of those who reported having had formal training in the use of herbs or other complementary therapies, more than 70% reported receiving more than three hours of such training; 22 participants (4%) had received 100 hours or more of such professional training in the past year.

Table 1
Table 1:
Baseline Demographic Characteristics and Scores for 537 Health Care Professionals Participating in an Internet Curriculum on Herbs and Other Dietary Supplements, 2000–2001

Although there were marked differences between different professional groups (e.g., higher knowledge scores for RDs compared with APNs and higher confidence scores for faculty compared with students), overall, the two groups had similar knowledge (mean = 9.75 of 20), confidence (mean = 4.2 of 10), and communication (mean = 1.4 of 4) scores at baseline in September 2000 (see Table 1).

Over the course of each group's participation, approximately 75 messages were posted to the listserv (averaging six to eight per week for each group). Fewer than 5% of the participants asked to un-enroll from the listserv.

Three fourths of the immediate intervention group and 95% of the waiting-list group completed the first follow-up questionnaire in December 2000. The immediate intervention group improved significantly more than did the waiting-list group on all three outcomes—knowledge, confidence, and communication practices (see Table 2). Even using the conservative intention-to-treat type of analysis, the outcomes significantly favored the immediate intervention group at p < .001 for knowledge and confidence and p < .01 for communication practices.

Table 2
Table 2:
Changes in Health Care Professionals' Knowledge, Confidence, and Communication Scores between Two Follow-up Surveys of an Internet Curriculum on Herbs and Dietary Supplements

Shortly after the waiting-list group received the curriculum and approximately four months after the immediate intervention group had completed it, two thirds of the immediate intervention group and 78% of the waiting-list group completed the second follow-up questionnaire (April 2001). Both groups scored significantly better than at baseline. The waiting-list group had a better improvement in knowledge scores in the raw data (see Table 2), but this difference was not statistically significant in the intention-to-treat analysis. Scores at the second follow-up were quite similar for the immediate intervention and waiting-list groups for confidence and communication both in the raw scores and in the intention-to-treat analysis (see Table 2). The changes in confidence over time for the immediate intervention and waiting-list groups are illustrated in Figure 1.

Figure 1
Figure 1:
Changes in health care professionals' confidence scores over the course of an Internet curriculum on herbs and dietary supplements. All participants were surveyed before the curriculum, after the immediate intervention (black bars), and after the waiting-list group's intervention (white bars).


This is the first study to describe the impact of an Internet-based curriculum on herbs and dietary supplements for health care professionals. Although we had hoped for 200 participants, over 500 professionals enrolled, and there was a very high rate of retention through the second follow-up, even in the absence of formal course credit, certification, or continuing education credits. Nearly all of the participants were able to complete the curriculum using the Internet, and very few un-enrolled from the moderated listserv discussion group. Furthermore, even in a professionally and geographically heterogeneous group of health care professionals, many of whom reported previous training in this topic, knowledge, attitudes, and self-reported practices were significantly improved through this Internet-based curriculum program. For example, an improvement of three points on the knowledge test represents an improvement of approximately one third of the average baseline score—a substantial as well as a statistically significant improvement. Furthermore, the participants were enthusiastic in their praise for the program, suggesting that it be offered again for continuing education credits and suggesting that, in the future, professionals should pay for the privilege of participating. These results are encouraging for medical educators seeking methods that are convenient and cost-effective for adult learners interested in topics that may not have been covered in traditional health education.

Despite the enormous growth in the use of herbs and dietary supplements in the United States, the traditional education of health professionals has not included information about these topics. Traditional curricula that are in-person or use paper and pencil, academic detailing, reminder systems, etc., have had inconsistent impacts on physicians' attitudes or actual skills.21,22 The Internet allows for providing education to busy professionals about novel therapies; a cursory search found over 100 continuing medical education Internet-based courses are available. In the last decade, Internet-based education and communication technology have become integral components of the medical school environment; they provide a unique opportunity to educate over great distances at times and locations that are convenient to the learners. Electronic education, including e-mail—based cases specifically, has proven acceptable and effective in improving participants' knowledge in dentistry, medicine, nursing, nutrition, osteopathy, pharmacy, and physical therapy.22,23 Participants can appreciate and learn from the curriculum even when they participate passively, simply reading the case and questions and waiting for the answers to be posted the following day. On-line case-based discussion groups have also proven acceptable and useful in continuing education for physicians. Although numerous electronic curricula have been developed and successfully implemented, we were unable to find any report about the effectiveness of educating health professionals about herbs or other topics in complementary medicine using the Internet.

Although the majority of U.S. medical schools offer some training about CAM, the courses are typically brief overviews and do not offer in-depth training or critical thinking about herbs and dietary supplements.24,25,26 Until existing faculty feel comfortable with the novel subject matter, integration of such topics into existing curricula is likely to be limited.27 The strengths of this study are that it included a large number of professionally and geographically diverse participants, an innovative curriculum-delivery system, and a conservative statistical analysis. The study design was more complex than the usual pre- and post-testing that is done with curriculum evaluation. We were concerned, however, because of the remarkable increase in information about herbs and dietary supplements provided by medical journals, that participants might be expected to have significant changes in knowledge, attitudes, and behaviors relative to this topic over time, and we wished to control for secular changes.

On the other hand, this study has several weaknesses. First, the sample was highly self-selected, which limits the generalizability of our findings. The RDs were not part of the planned sample and were likely to have had more previous education about herbs and dietary supplements than most random samples of clinicians. The knowledge, attitudes, and practices of the final sample are likely to represent those professionals most keenly interested in and self-educated about herbs and dietary supplements. They may have been more likely than others to have continued to participate, to refer to the Internet-linked sites, and to have discussed the course content with colleagues. This difference, however, is likely to have biased our study against finding any impact from the curriculum. The sample contained few men and few professionals from racial and ethnic minorities, who might have had substantially different views of herbs and dietary supplements. The sample was highly electronically literate, having completed the baseline questionnaires (and the subsequent educational program) nearly entirely on the Internet.

Second, the study's instruments, although they had face validity and underwent substantial testing and revision prior to implementation, have not been validated against a “gold standard.” The knowledge and confidence scales had reasonable internal validity as measured by Cronbach's alpha, but the communication scale was not as internally consistent and needs revision. To our knowledge, however, there is no existing “gold standard” against which to compare test questions regarding knowledge, confidence, or communication about herbs and dietary supplements. The study used self-report rather than observation of clinical practice, and the extent to which self-report on a novel questionnaire correlates with actual clinical practice is unknown.

Third, this study did not compare different ways of educating health professionals to better advise patients and families about herbs and dietary supplements. Although the participants cited more previous training in the use of herbs and supplements than we had anticipated, we did not ascertain which educational format they found most useful. Future studies will need to address the effectiveness and costs of various curricular components. For example, is it necessary to include the listserv discussion group? Could the modules be delivered over a shorter period of time rather than spacing them out over ten weeks? Are the Internet links necessary or would more links be helpful? How much would a traditional written syllabus enhance learning?

These results have important implications for health care delivery, professional education, and future research. Health policies should be developed to ensure that the geographically and professionally diverse health care professionals have sufficient knowledge and confidence to counsel patients about commonly used herbs and dietary supplements. Studies are needed to assess the impacts of educational programs on clinicians' actual behaviors in talking with patients, noting patients' uses of herbs and dietary supplements in the health record, and reporting adverse effects to appropriate authorities. Future studies will need to include more representative samples of clinicians nationwide, find ways of validating the assessment tools, and assess the most cost-effective strategies for educating clinicians about herbs and dietary supplements. Such studies could become models for assessing and educating diverse groups of clinicians about other novel therapies or topics. Additional studies might target the impact of such training on patients' satisfaction and compliance with usual health recommendations. These data represent an essential step for this field of educational inquiry.


1. Eisenberg DM, Davis RB, Ettner SL, et al. Trends in alternative medicine use in the United States, 1990–1997: results of a follow-up national survey. JAMA. 1998;280:1569–75.
2. Elder NC, Gillcrist A, Minz R. Use of alternative health care by family practice patients. Arch Fam Med. 1997;6:181–4.
3. Gulla J, Singer AJ, Gaspari R. Herbal use in ED patients. Acad Emerg Med. 2001;8:450.
4. Ottolini M, Hamburger E, Loprieto J, et al. Alternative Medicine Use among Children in the Washington, DC Area. Ambulatory Pediatr. 2001;2:122–5.
5. Weiss SJ, Takakuwa KM, Ernst AA. Use, understanding, and beliefs about complementary and alternative medicines among emergency department patients. Acad Emerg Med. 2001;8:41–7.
6. Breuner CC, Barry PJ, Kemper KJ. Alternative medicine use by homeless youth. Arch Pediatr Adolesc Med. 1998;152:1071–5.
7. Fairfield KM, Eisenberg DM, Davis RB, Libman H, Phillips RS. Patterns of use, expenditures, and perceived efficacy of complementary and alternative therapies in HIV-infected patients. Arch Intern Med. 1998;158:2257–64.
8. Kemper K, Wornham W. Consultations for holistic pediatrics for inpatients and outpatient oncology patients at a children's hospital. Arch Pediatr Adolesc Med. 2001; in press.
9. Richardson MA, Sanders T, Palmer JL, Greisinger A, Singletary SE. Complementary/alternative medicine use in a comprehensive cancer center and the implications for oncology. J Clin Oncol. 2000;18:2505–14.
10. Allen R, Cushman LF, Morris S, et al. Use of complementary and alternative medicine among dominican emergency department patients. Am J Emerg Med. 2000;18:51–4.
11. Maskarinec G, Shumay DM, Kakai H, Gotay CC. Ethnic differences in complementary and alternative medicine use among cancer patients. J Altern Complement Med. 2000;6:531–8.
12. Pachter LM, Sumner T, Fontan A, Sneed M, Bernstein BA. Home-based therapies for the common cold among European American and ethnic minority families: the interface between alternative/complementary and folk medicine. Arch Pediatr Adolesc Med. 1998;152:1083–8.
13. Eisenberg DM, Davis RB, Ertner SL, et al. Unconventional medicine in the United States: prevalence, costs and patterns of use. N Engl J Med. 1993;328:246–52.
14. Hensrud DD, Engle DD, Scheitel SM. Underreporting the use of dietary supplements and nonprescription medications among patients undergoing a periodic health examination. Mayo Clin Proc. 1999;74:443–7.
15. Sibinga E, Ottolini M, Duggan A, Wilson M. Communication about complementary/alternative medicine use in children. Pediatr Res. 2000;47:226A.
16. Committee on Children with Disabilities. Counseling families who choose complementary and alternative medicine for their child with chronic disease or disability. Pediatrics. 2001;107:598–601.
17. Kemper JK. Integrative medicine: talking with families about complementary, alternative, and mainstream medical therapies in acute care settings. Office and Emerg Pediatr. 2000;13:45–9.
18. Moore C, Adler R. Herbal vitamins: lead toxicity and developmental delay. Pediatrics. 2000;106:600–2.
19. Fessenden JM, Wittenborn W, Clarke L. Gingko biloba: a case report of herbal medicine and bleeding postoperatively from a laparoscopic cholecystectomy. Am Surg. 2001;67:33–5.
20. Kemper K, Vincent E, Scardapane J. Teaching an integrated approach to complementary, alternative and mainstream therapies for children: a curriculum evaluation. J Alternat Complement Med. 1999;5:261–8.
21. Barnes BE. Creating the practice-learning environment: using information technology to support a new model of continuing medical education. Acad Med. 1998;73:278–81.
22. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA. 1992;268:1111–7.
23. Marshall JN, Stewart M, Ostbye T. Small-group CME using e-mail discussions. Can it work? Can Fam Physician. 2001;47:557–63.
24. Carlston M, Stuart MR, Jonas W. Alternative medicine instruction in medical schools and family practice residency programs. Fam Med. 1997;29:559–62.
25. Sampson W. The need for educational reform in teaching about alternative therapies. Acad Med. 2001;76:248–50.
26. Wetzel MS, Eisenberg DM, Kaptchuk TJ. Courses involving complementary and alternative medicine at US medical schools. JAMA. 1998;280:784–7.


Example Module: Echinacea

CASE: The mother of a two-year-old with recurrent colds and ear infections asks you if echinacea will help prevent these infections and if it's safe for young children.


Which of the following is true about echinacea?

  • _____ a. Studies from several randomized, controlled clinical trials suggest that taking echinacea at the first sign of a cold can help reduce the duration and severity of cold symptoms.
  • _____ b. Controlled trials in Germany have shown that echinacea is helpful in reducing the number of colds for children under 13 years old.
  • _____ c. Echinacea needs to be given for at least eight weeks to have any benefits on immune function.
  • _____ d. Due to numerous reports of side effects, several states have limited sales of echinacea products.


Answer A is correct. European studies in vitro, in animals, and in humans support the use of echinacea in treating the common cold in adults. However, studies do NOT consistently support the use of echinacea as an effective prophylactic agent for upper respiratory infections. Furthermore, randomized, controlled trials have NOT been reported for children. Effects on immune function in adults appear quickly; most herbalists recommend that echinacea not be used for more than eight to 12 weeks. Although echinacea is widely used in Europe and the U.S., reports of toxic side effects (including allergic reactions) are quite rare.

Want to know more about echinacea?

Just click on the web address below to link directly with Longwood Herbal Task Force resources:

Echinacea monograph:

Echinacea clinician summary:

Echinacea patient information handout:

27. Kligler B, Gordon A, Stuart M, Sierpina V. Suggested curriculum guidelines on complementary and alternative medicine: recommendations of the Society of Teachers of Family Medicine Group on Alternative Medicine. Fam Med. 2000;32:30–3.
© 2002 Association of American Medical Colleges