Secondary Logo

Journal Logo

Evaluation of an Electronic Consultation Service in Obstetrics and Gynecology in Ontario

Shehata, Fady MBBCh, MSc; Posner, Glenn MDCM, FRCSC; Afkham, Amir BEng; Liddy, Clare MD, MSc; Keely, Erin MD, FRCPC

doi: 10.1097/AOG.0000000000001431
Contents: Original Research
Journal Club

OBJECTIVE: To describe the effectiveness of an electronic consultation (eConsult) service by examining the number of traditional referrals that were avoided as a result of the service, to characterize the type and content of the clinical questions being asked, and to describe the time required for the specialist to complete each eConsult.

METHODS: This is a retrospective electronic chart review study. All eConsults directed to obstetrics and gynecology from July 2011 to January 2015 were reviewed. Each eConsult was categorized by clinical topic and question type in predetermined categories. Mandatory post-eConsult surveys for primary care providers were analyzed to determine the number of traditional consults avoided and to gain insight into the perceived value of eConsults. The amount of time reported by the specialist to answer each eConsult was analyzed.

RESULTS: A total of 394 of 5,597 eConsults were directed to obstetrics and gynecology (7.0%). In 34.3% of eConsults, primary care providers indicated that a traditional consult was avoided. Pregnancy issues and gynecologic cancer screening issues were the most common queries. Primary care providers highly valued the eConsult and the majority of eConsults were completed within 15 minutes (98.8%).

CONCLUSION: Electronic consultations were effective at reducing the number of traditional consults requested over 3.5 years. This initiative has potential to reduce current wait times for traditional consultation in Canada and to make the consultation process more effective. The service was feasible and well-received by primary care providers.

An electronic consultation service in Ontario is effective in reducing the number of traditional consults requested by primary care providers and is well-received.

Departments of Obstetrics, Gynecology and Newborn Care and Family Medicine, University of Ottawa, the Champlain Local Health Integration Network, the C. T. Lamont Primary Healthcare Research Centre, Bruyère Research Institute, the Division of Endocrinology/Metabolism, the Ottawa Hospital, and the Ottawa Hospital Research Institute, Ottawa, Ontario, Canada.

Corresponding author: Glenn Posner, MDCM, MEd, Department of Obstetrics, Gynecology and Newborn Care, University of Ottawa, The Ottawa Hospital, General Campus, 501 Smyth Road, 8th Floor, Room 8303 (Box 804), Ottawa, ON K1H 8L6, Canada; e-mail:

Supported by the Ministry of Health and Long-Term Care, the Canadian Institutes of Health Research, The Ottawa Hospital Academic Medical Organization Innovation Fund, e-Health Ontario, the University of Ottawa Department of Medicine, Bruyère Research Institute, and the Champlain Local Health Integration Network.

Financial Disclosure Mr. Afkham is an employee of the Local Health Integration Network (LHIN). The other authors did not report any potential conflicts of interest.

The number of Canadians being referred to a specialist increases yearly.1 In Ontario, approximately 54,000 consults are referred to specialists each day.2 This has dramatically extended the wait time to see a specialist in Canada from 3.7 weeks in the year 1993 to 8.5 weeks by 2014.3 Compared with 10 other developed countries, Canada holds the second longest average wait time to see a specialist.4,5 In gynecology, the average wait time to see a specialist is 8.7 weeks and from specialist to treatment is another 7.4 weeks, making a total of 16.1 weeks from the time a referral is made to a gynecologist to the time the patient receives treatment.3

In an effort to improve access to care, alternatives to traditional consultation have been explored such as telephone consultation and e-mail consultation. However, each alternative has its own limitations including the need for health care providers to be available at the same time and inability to meet privacy and security standards, respectively.6,7 As a result of these limitations, electronic consultation (eConsult) services have been developed that allow primary care providers (family physicians and nurse practitioners) to electronically submit a consult to a specialist who may be able to provide advice without the need for a face-to-face consultation.6,8 This approach has been implemented successfully in several jurisdictions in other parts of the world, for example, New Zealand and the United States.8–10

This study aims to analyze all the obstetrics and gynecology eConsults, which were completed from July 2011 to January 2015. Specifically, we aim to estimate the effectiveness of the eConsult service by number of traditional referrals that were avoided as a result of the eConsult service and health care provider satisfaction. The secondary objective for this study is to characterize the questions included in each eConsult by quantifying the most common clinical topics and types of questions asked to better understand what type of questions are most amenable to eConsultation. In addition, to describe the effects on workload, the time taken to finish each consult by the specialist is also explored.

Back to Top | Article Outline


The project was approved by the Ottawa Health Science Network research ethics board. A secure, encrypted database containing all eConsults completed between the periods of July 2011 to January 2015 was created and maintained by the project manager (A.A.). Access to the database was limited to study investigators. All eConsults were answered by a single Royal College of Surgeons of Canada–certified obstetrician–gynecologist (ob-gyn) who had been in independent practice for 7 years at the start of the project.

Using a secure, web-based tool, primary care providers direct specific patient questions to a specialty, in this case obstetrics and gynecology. The primary care provider has the opportunity to append diagnostic images, reports, pictures, or any other information that can aid the specialist in understanding the problem or the reason for consultation. The case is then assigned to a specialist, in this case a single ob-gyn participating who was participating in the project at the time, resulting in a notification e-mail to that specialist with the expectation that it will be completed within 7 days. This eConsultation takes place between two physicians with no contact between the specialist and the patient. The specialist reads and responds to the case in an online browser. From a medicolegal perspective, the specialist must indicate that they have enough information to answer the question. They also have an opportunity to ask for additional information or suggest that the patient be referred for a face-to-face consultation. Once completed, the primary care provider shares the opinion of the consultant with the patient at their next appointment.

A predefined list of clinical topics (reached by consensus between authors) and types of questions (based on validated taxonomy) asked by the primary care providers was created. A total of 23 different clinical topics were included and are listed in (Table 1). We chose to group all topics related to pregnancy together because it was difficult to separate them into different topics. All clinical topics with less than 10 consults were grouped separately into “other topics” to facilitate analysis. There were six different categories of types of questions asked by the primary care providers (Fig. 1).

Table 1

Table 1

Fig. 1

Fig. 1

This study was a retrospective electronic chart review study. All eConsults within the defined study period (394 eConsults) were reviewed and categorized retrospectively by clinical topic and type of question by one rater (F.S.). To ascertain agreement on the categorization of each eConsult, a random selection of 54 eConsults (approximately 14% of the total) was independently reviewed by the specialist (G.P.). Disagreements (five eConsults) were mainly regarding clinical topic chosen and were resolved through rereviewing and discussing the eConsult in detail until agreement was achieved between the lead author and the specialist. All data were exported into an Excel database for analysis. Descriptive analyses were used to quantify the most common clinical topics or question types.

After each eConsult, a mandatory survey is completed by the primary care provider. The primary care provider could not close a case and receive a transcript of the consultation without completing the survey. The survey was comprised of five questions (Box 1). Questions ranged from assessing the value of the eConsult service to determining whether a referral was avoided as a result of the eConsult.

Back to Top | Article Outline

Box 1 Post–Electronic Consultation Survey Detailed Questions and Answers Cited Here...

  • Q1. Which of the following best describes the outcome of this eConsult for your patient:
    • 1. I was able to confirm a course of action that I originally had in mind
    • 2. I got good advice for a new or additional course of action
    • 3. I did not find the response very useful
    • 4. None of the above (please comment)
  • Q2. As a result of this eConsult, would you say that:
    • 1. Referral was originally contemplated but now avoided at this stage
    • 2. Referral was originally contemplated and is still needed - this eConsult likely leads to a more effective visit
    • 3. Referral was not originally contemplated and is still not needed - this eConsult provided useful feedback/information
    • 4. Referral was not originally contemplated, but eConsult process resulted in a referral being initiated
    • 5. There was no particular benefit to using eConsult in this case
    • 6. Other (please comment)
  • Q3. Please rate the overall value of the eConsult service in this case for your patient:
    • Minimal 1 2 3 4 5 Excellent
  • Q4. Please rate the overall value of the eConsult service in this case for you as a primary care provider:
    • Minimal 1 2 3 4 5 Excellent
  • Q5. We would value any additional feedback you provide

eConsult, electronic consultation.

Reprinted from Liddy C, Maranger J, Afkham A, Keely E. Ten steps to establishing an e-consultation service to improve access to specialist care. Telemed J E Health 2013;19:982–90.

Two time elements were analyzed; first, the amount of time between the primary care provider's submitting the eConsult until an answer was received (response time) and, second, the amount of time reported by the specialist to complete the eConsult. The specialist was given four timeframes to choose from: less than 10 minutes, 10–15 minutes, 15–20 minutes, and more than 20 minutes. Initially, these data were collected for the purposes of remunerating physicians for their time but they now serve as an outcome of interest in terms of the time commitment and feasibility of the service.

Back to Top | Article Outline


Of the 5,597 eConsults submitted to the service, 394 (7.0%) were directed to ob-gyns. Questions were submitted by 151 primary care providers—126 medical doctors and 25 nurse practitioners—91% with urban practices and 9% with rural practices. The mean age of patients included was 43.2±15.4 years (range 8–95 years old).

Pregnancy issues and gynecologic cancer screening issues were the most common topics in our study, accounting for 18% each (Fig. 2). Pregnancy issues included infertility, preconception planning, antepartum bleeding, antenatal screening, infections in pregnancy, medication safety as well as other symptoms and complaints of pregnancy. Obstetric concerns were grouped together because there were low numbers in each category and most related to antenatal care. Gynecologic cancer screening topics included abnormal Pap tests, endometrial hyperplasia, and postmenopausal bleeding. Vulvovaginal symptoms comprised 13%, abnormal uterine bleeding 12%, contraception 10%, menopause 9%, abnormal pelvic ultrasonography 8%, and uterine fibroid 3%. All eConsult topics that occurred less than 10 times were grouped into a separate category, “other.” Other topics included vulvovaginal symptoms (13%), abnormal uterine bleeding (12%), contraception (10%), menopause (9%), abnormal pelvic ultrasonography (8%), and uterine leiomyoma (3%).

Fig. 2

Fig. 2

Questions about drug treatments and general management questions were the most common question types in our study, comprising 28% and 27%, respectively. Drug treatment questions included inquiries about the drug of choice for a specific condition, safety of the drug in pregnancy as well as how specific drugs are usually prescribed (Fig. 1).

In the post-eConsult surveys, which were all completed in full, primary care providers were asked whether referral was originally contemplated but avoided as a result of this eConsult. In 135 of 394 (34.3%) of eConsults, a referral was originally contemplated but now avoided as a result of the eConsult. In 165 of 394 (41.9%) of cases, a referral was not planned but the eConsult provided useful feedback. In 73 of 394 (18.5%) of cases, referral was still needed based on specialist advice, but the eConsult resulted in a more effective visit, because the specialist provided recommendations for necessary workup before referring the patient. A second question in the post-eConsult survey asked primary care providers to evaluate the value of the eConsult for the patient. In more than 50% of cases, primary care providers got good advice for a new or additional course of action; in 46.7% of cases, primary care providers indicated that the eConsult helped them to confirm their existing plan of action or treatment.

The vast majority of primary care providers rated the overall value of the eConsult service very highly. On a scale from 1 (minimal) to 5 (excellent), 93.4% of primary care providers gave the service a rating of 4 or 5 on its overall value for patients, and 94.2% gave it a rating of 4 or 5 on its value for them as health care providers. Examples of comments provided by the primary care providers included: “clear response and good teaching for future cases,” “efficient, speedy, and reassuring. My patient was surprised about the technology and how quickly a response was obtained,” “fast and effective way of communicating with a specialist,” “having the opinion of the specialist helped in reassuring me and the patient about the next course of action,” “I have used eConsult several times now and find it extremely efficient and a resource that we are so lucky to have in our day-to-day practice,” “my patients are appreciating the quick replies to the eConsults,” “this eConsult provided an immediate plan of care, which both I and my patient were very pleased with,” “this is the perfect use of eConsult. Avoided a consult and helped adjust my overall practice in this situation based on the advice provided (ie, I was overscreening),” “timely answer. Great feedback. Offered useful resources.”

The vast majority of the consults required less than 10 minutes (80.5%) for the ob-gyn to complete, and nearly all of them were completed within 15 minutes (98.8%). Most of the remaining eConsults were completed within 20 minutes (Fig. 3).

Fig. 3

Fig. 3

Back to Top | Article Outline


This study reports on a secure, web-based eConsult service in obstetrics and gynecology. This highly valued, efficient system demonstrated change in care for patients by reducing the need for face-to-face referrals and providing advice that resulted in a new course of action by the primary care provider. Evidence for this change in referral practice from evaluation data is a surrogate outcome based on self-report by primary care providers, but numbers of actual consultations avoided in our province are not measurable. Moreover, the eConsult project would still be a success if the absolute number of consultations was not reduced but the actual consultations initiated were more efficient (ie, correct investigations ordered before the visit) or if patients who truly require face-to-face consultations were being seen. This may have vast implications for the future as e-health becomes more integrated into our health care system.

In this project, the eConsultant is paid a stipend based on the amount of time they report spending on the eConsult. From a practical perspective, could there be pushback from specialists who enjoy billing for easy consults? Certainly—but we believe that it is incumbent on physicians to try to be part of the solution to our health care woes, one of which being timely access to care for patients who really need it. We will all benefit from a more efficient and optimized health care delivery system.

A measure of the effectiveness of the eConsult service can be stipulated through the one-on-one teaching that occurs in each eConsult. It can also be drawn from the satisfaction reported by the primary care providers. In addition, the eConsult system provided a platform in which the primary care provider can communicate ultrasound reports, images, or laboratory values to the specialist. This could not have been possible if a phone consultation system was used.

Analysis of the types of questions and clinical topics received is a unique opportunity to understand clinical scenarios that primary care providers have questions about. This information could be used to inform planning of continuing medical and professional development events for primary care providers. Interestingly, primary care providers asked more questions regarding pregnancy issues than any other topic. In the gynecologic cancer screening topic, questions about endometrial hyperplasia and the normal width of the endometrium on ultrasonography were very common and for each eConsult on that topic, the specialist forwarded the specialty society guidelines on that topic in an effort to provide education. Therefore, eConsults are not only an efficient way of consulting a specialist, but may be used as a forum to share knowledge and increase the capacity of primary care providers to manage these types of clinical problems independently. Future research is needed to help determine whether focused, directed primary care provider education through eConsults or other focused medical education sessions based on the content of eConsults improves overall primary care provider knowledge in obstetrics and gynecology.

The eConsult service changed primary care provider referral patterns and the clinical course of action. The proportion of all cases in which referral was originally contemplated but now avoided as a result of the eConsult advice, 34%, is in a similar range as the average across all specialties within the Champlain BASE service (40%).11 If this service were more broadly available, there are huge potential savings for our health care system, both in terms of avoided face-to-face referrals, and also in terms of proving more timely care for patients and potentially reducing risks of further degradation or complication. In an economic analysis of the overall Champlain BASE service, factoring in its cost and the cost of traditional referral prevented, is expected to break even after approximately 7,800 eConsults.11

Our study showed that primary care providers highly valued the eConsult service. In 50.8% of the time, a suggestion for a new or additional course of action was received as a result of the eConsult regardless of whether a traditional referral was eventually recommended. In 46.7%, it confirmed an action that they already had in mind, which may help reassure the primary care provider and the patient that they are on the right path. The eConsult service was recognized as a valuable educational tool by primary care providers because they were more engaged in patient care through participating in the thought process involved in each consult.

Finally, it is important to notice that the eConsult service did not consume much of the specialist's time because it took less than 15 minutes to finish 98.8% of eConsults in our study, and in fact, the vast majority (80.5%) of eConsults were completed in less than 10 minutes. This is important because it can be used to educate new specialists who are interested to join the project and give them an estimate of how much time they need to commit in their schedule for this kind of service. It also provides useful information for deciding on the best payment models.

One limitation of our study is that all eConsults were answered by a single gynecologist. Potentially, the feedback for this one gynecologist's consultations might not be reflective of the service that would be provided by multiple gynecologists. However, the Champlain BASE service has processed more than 10,000 eConsults with similar results across all specialty services.

The eConsult service for obstetrics and gynecology has been successfully implemented in our region and addresses a wide range of clinical topics. It has the potential to reduce both the wait times to see a specialist and the frustrations of patients and primary care providers in our current congested health care system.

Back to Top | Article Outline


1. Carrière G, Sanmartin C. Waiting time for medical specialist consultations in Canada, 2007. Health Rep 2010;21:7–14.
2. Manuel DG, Maaten S, Thiruchelvam D, Jaakkimainen L, Upshu R. Primary care in the health care system. In: Jaakkimainen L, Upshur R, Klein-Geltink JE, Leong A, Maaten S, Schultz SE, et al., editors. Primary care in Ontario: ICES atlas. Toronto (Canada): Institute for Clinical Evaluative Sciences; 2006.
3. Fraser Institute. Waiting your turn: wait times for health care in Canada, 2014 report. Available at: Retrieved December 27, 2015.
4. Berthelot JM, Sanmartin C. Access to health care services in Canada, January to December 2005. Ottawa (Canada): Statistics Canada; 2006.
5. Fraser Institute. Waiting your turn: wait times for health care in Canada, 2011 report. Available at: Retrieved April 11, 2016.
6. Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building access to specialist care through e-consultation. Open Med 2013;7:e1–8.
7. Keely E, Liddy C, Afkham A. Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health 2013;19:733–8.
8. Liddy C, Maranger J, Afkham A, Keely E. Ten steps to establishing an e-consultation service to improve access to specialist care. Telemed J E Health 2013;19:982–90.
9. Straus SG, Chen AH, Yee H Jr, Kushel MB, Bell DS. Implementation of an electronic referral system for outpatient specialty care. AMIA Annu Symp Proc 2011;2011:1337–46.
10. Gray BH, Bowden T, Johansen I, Koch S. Issues in international health policy: electronic health records: an international perspective on “meaningful use.” Issue Brief (Commonw Fund) 2011;28:1–18.
11. Liddy C, Deri Armstrong C, Drosinis P, Mito-Yobo F, Afkham A, Keely E. What are the Costs of Improving Access to Specialists through eConsultation? The Champlain BASE Experience. Stud Health Technol Inform 2015;209:67–74.


© 2016 by The American College of Obstetricians and Gynecologists. Published by Wolters Kluwer Health, Inc. All rights reserved.