The specialty of obstetrics and gynecology was the first to experiment with an on-line residency application in cooperation with the Association of American Colleges.1 All specialties, except those in the early match, now participate in the Electronic Residency Application Service (ERAS). The advent of the internet with high-speed capabilities makes the application and selection process quick and efficient for both programs and applicants. Residencies can now select applicants using filters in ERAS and invite them automatically by e-mail with a few clicks in the software.
Scheduling resident applicants for an interview is a time-consuming process that typically entails hours on the telephone and frequent e-mails in an exchange encumbered by time zone differences, after-hours availability of personnel, and the wait time candidates experience in getting official notification of the scheduled date and time.
In 2005 we designed an on-line scheduler for faculty to schedule themselves for applicant interviews. We discovered that a great deal of time and effort was saved using an on-line scheduler. All the faculty interview positions were filled that year without a single telephone call to faculty. Faculty that procrastinated received follow up e-mail reminders to schedule themselves.
Faced with the challenge of a staffing shortage in the summer of 2006, we decided to create an on-line scheduler for applicants to view the available dates, pick a date, and receive an immediate confirmation by e-mail. We anticipated that an on-line applicant scheduler would decrease the burden on our reduced office staff to schedule invitees. The same computer programmer who created the faculty schedule was recruited from the hospital’s information service to design the applicant scheduler. We surveyed the applicants who used the scheduler to assess their views of the feasibility and acceptability of on-line scheduling.
MATERIALS AND METHODS
The programmer created the applicant scheduler in ColdFusion (Adobe Systems Incorporated, San Jose, CA).2 The features of the scheduler included access to the hospital’s intranet, a listing of available interview slots, a point-and-click hyperlink to reveal these dates, and immediate confirmation followed by a reminder e-mail 1 week before the scheduled interview date. The administrative interface made it simple to enter the ERAS numbers, names, and e-mail addresses of the invitees. It was also easy to edit the dates, correct or update applicant data, and to monitor slots filled and unfilled by invitees. We scheduled a total of 10 days, divided into morning and afternoon sessions, with four invitees per session to ultimately accommodate 80 invitees. We designed the scheduler to delete invitees from its database if they failed to schedule themselves within 5 days of our invitation. We tested the program’s function and utilities several times before the launch date.
We selected applicants for interviews through ERAS. We sent them a scripted e-mail invitation from ERAS that included the hyperlink to the scheduler Web site. The e-mail advised invitees to call the department if they encountered any problem with the scheduler. It also informed them that they had 5 days to schedule on-line and that after that period we would withdraw the invitation. We sent them a reminder e-mail after 3 days of the invitation.
After accessing the site, applicants entered their ERAS numbers to open the scheduler. Similar to reserving an airline seat on a flight, invitees selected a date from the available list without seeing the names of the scheduled applicants. The program gave them immediate confirmation that they were scheduled for the selected date. We used the scheduler to book 71 of the 80 listed available slots and scheduled an additional 12 applicants by e-mail or telephone to give us some flexibility in scheduling late additions or for filling cancellations.
We e-mailed a 7-item questionnaire to the 71 invitees who booked on-line after scheduling. The e-mail survey asked them if they booked on-line with our program, booked on-line with other residency programs, their opinion of the scheduler (liked it or not and why), and if they preferred booking on-line, found it easier, and offered them more flexibility than with traditional methods. It asked them to indicate whether booking on-line affected their views of the quality of a residency program. They were told that their responses would not affect their interview status or outcome, and that the program director did not have access to the individual responses. We followed up with two e-mail reminders to those who were slower to respond. We received approval from the institutional review board of the Women and Infants Hospital to conduct the survey. We did not survey the remaining 12 applicants we scheduled by e-mail or telephone.
We calculated percentages and the confidence intervals from the survey data. We used a free on-line program from Dimension Research, Inc. (Lombard, IL) to calculate the confidence intervals around the proportions.3
We invited 54 applicants in the first wave of invitations. Of these 97% (95% confidence interval [CI] 90.7–101.2) booked themselves within 3 days of our e-mail and two withdrew their applications. We sent a series of invitations until we reached our quota of filled positions. We scheduled a total of 83 applicants, of which 86% (71 of 83) were invited to book themselves on-line and did so. We booked the remaining 12 as fill-in positions either by e-mail or telephone invitation after on-line booking had been completed.
All scheduled applicants received our e-mail questionnaire. We followed up with three e-mail reminders until all 83 (100%) invitees responded to the questionnaire. Sixty-three percent responded to our first request, 33% to the second request, and 5% to the third request. We excluded from the survey the 14% (12 of 83) who we scheduled by e-mail or telephone, yielding data from 71 respondents’ questionnaires for analysis.
Sixty-five of the 71 (92%, 95% CI 85.4–98.6) said they liked the scheduling program. Most of the 71 respondents (93%; 95% CI 85.7–98.3) indicated that ours was the only residency program using an on-line scheduler; the remaining five respondents reported that they booked their interviews on-line with other programs. A follow-up e-mail inquiry revealed that these applicants could not recall the names of these residency programs.
Ninety-two percent (95% CI 84.3–97.7) liked using the on-line scheduler, whereas 8% (95% CI 9.0–14.9) did not. Asked of their booking preferences 77% (95% CI 67.2–86.8) preferred to book on-line, 7 (95% CI 1.07–12.9) percent preferred to schedule by telephone or e-mail, and 15% (95% CI 6.7–23.3) had no preference. Most (80%, 95% CI 70.7–89.3) found the scheduler easier to use, and the remaining 20% (95% CI 9.9–28.1) considered it equivalent to traditional scheduling methods. Fifty-four invitees gave a rationale for why they thought the program was easier to use. Many liked the ability to see the available dates and to get quick confirmation—“I knew immediately that the date I preferred was available and confirmed, instead of responding to the program coordinator and having to wait to hear back to be sure that I got the date I wanted. This helped when trying to coordinate a number of interviews in a short period of time.” Others liked the ability to schedule at their convenience—“It was convenient, particularly for after-hours scheduling. I liked that it allowed you to see which days were filled already,” “Easy to use and able to schedule my interview at my own convenience.” Those preferring traditional methods to the scheduler liked dealing with a contact; eg, “although convenient, it’s rather impersonal. I would prefer to interact with someone via phone or e-mail. It was easy though.” or they would have liked more scheduling flexibility; eg, “I would have liked to be able to change my date or see which days were still open to schedule on after scheduling my initial date.”
Respondents were largely split over the question of which method offered the most flexibility in scheduling interviews. Of the 70 respondents, 40% (95% CI 28.5–51.5) thought the scheduler offered more flexibility, 31% (95% CI 20.2–41.8) thought traditional methods were more flexible, and 29% (95% CI 17.5–38.5) felt that the two methods were comparable in flexibility.
Because of the novelty of the scheduler, we wanted to see whether having an on-line scheduler affected invitees’ opinions of the quality of the residency program. Of the 71 respondents, 72% (95% CI 60.5–81.6) felt it made no difference, whereas 28% (95% CI 17.7–38.4) thought it did reflect on the quality of the program. Although the majority felt it did not make a difference in their opinion of the program, they did concede that it reflected on the program’s use of technology, as reflected in this statement: “(the) mode of scheduling shouldn’t affect teaching or medical care, although it does show an increased use of progressive technology which I think is a positive addition to any program,” and in this comment: “while having an organized system for scheduling shows that the program is organized, my overall opinion will be based on the reputation, surgical training, and resident satisfaction.” Those who felt the scheduler affected their opinion of the program’s quality commented too on the use of technology, and added that they felt that it reflected on the program’s efficiency, organization, and respect for the invitees’ time. Their comments included: “It showed me that they were quite organized and were also trying to make this as easy as possible for the applicants”; “It shows me that technology and forward thinking is important to the program—and this is something that is very important to me”; “It was nice to see (the program) trying something new and it made me excited to see what else was going on at the program.”; and “It communicates that the program has been thoughtful about the systems they create, and they have a plan.” One invitee in this group commented “I felt slightly intimidated by the program because of this (scheduler).”
We were delighted with the response from the applicants. We sent the majority (54/80) of our invitations out in one e-mail and within a few days all the applicants had either scheduled or withdrawn their request for an interview, all without a single telephone call or e-mail from the invitees. The applicant scheduler worked well. We informed applicants to notify us if they had any difficulties with the scheduler. Two applicants picked the wrong time slot for a given day and were unable to make a switch. We received their e-mail request for assistance and we quickly changed their assignments using the administrative functions of the program. Invitees did not request other changes even with our offer for off-line assistance in scheduling.
Our survey results showed that most invitees liked using the on-line scheduler primarily for its ease of use, immediate confirmation, and the ability to arrange their interview date at any time, without depending on a response from another person to confirm the appointment. They liked using technology to make their scheduling easier and convenient. The scheduler did not affect the invitees’ views about the quality of the residency program for the most part, however many felt that the scheduler reflected on the program’s innovativeness and efficiency.
We recognize that this study has limitations. Invitees may report what they think we want to see due to the request coming from the residency program. To help reduce this type of error, we used several measures. We had the medical education office secretary send the requests and follow-up reminders, we told invitees that the program director would not see their responses, that the summary data would not use identifiers, and that their responses had no bearing on their interview status or review. We were persistent in getting all the on-line scheduled applicants to respond (because their responses were not anonymous), and such persistence could produce bias in both directions. This descriptive study limits our ability to generalize beyond our invitees’ responses to infer that all applicants would like other residency programs to have an on-line scheduler.
There are potential downsides to using an on-line scheduler. For those applicants who receive later invitations, it is obvious to them that they have fewer available options and that they received a lower priority on the invitation list. It would be like seeing that the good seats are already taken for a flight or concert. These applicants might feel less wanted by the program and carry that feeling with them into the interview and beyond. To help reduce this potential limitation, we plan to have filled dates disappear from the on-line selection screen such that the later invitees do not see them. We also plan to configure the program with a message stating that they have chosen a particular date and time, and give them the option to accept or cancel the selection. This will help preclude a scheduling error. Some invitees would still prefer to schedule their interview with a representative from the program through e-mail or telephone. Some of these invitees may perceive they can negotiate a better schedule through personal contact.
We plan to modify and use the on-line scheduler for next interview season. We would encourage other residency programs to consider implementing similar scheduling methods. The on-line scheduler saved hours of staff time, at a time when we had no one available to do the scheduling. It is an efficient, convenient and labor-saving technological tool.