Almost all medical institutions host websites with specific content for different audiences. Commercial websites tend to analyze their target audiences to modify the website for maximizing revenue. The revenue in medical institutions is not measured in e-commerce values. Therefore, their analyses of the user data will differ from established commercial approaches. However, it is important to assess user data, because unused content should either be modified to meet the demands of the targeted audience or removed to reduce the time and effort for website maintenance. Medical institutions’ websites usually target 4 specific audiences: patients, doctors, medical students, and job applicants. The needs of those 4 groups reveal important differences. Patients usually use information on the web to gain insight into their diseases and treatments. They may choose their healthcare provider according to the information they find,[1,2] request appointments online, use telemedicine services, or are interested in wearable health devices with transmission of medical information for smart healthcare purposes.[4–6] For referring physicians, institutional websites offer information on how to refer patients. Medical students are usually able to obtain information about the teaching schedules and examinations on institutional websites. Also, information regarding job applications is also presented online providing detailed instructions on the format for submitting an application.
One main focuses of recent studies is the usability of hospital websites and on what constitutes suitable content.[7–9] Other studies analyze user behaviors of patients or test persons in experimental settings.[1,10] Himmel et al took a different approach via an online questionnaire presented to website users. Another approach is to assess the use of website access data to gain insight on how website visitors utilize and interact with the website.[12,13]
The goal of our study was to analyze the utility of the website of a large tertiary referral center in ophthalmology with regard to the different target audiences (patients, medical students, applicants, referring ophthalmologists). We aim to draw conclusions on how all 4 groups utilize the content on offer. Since we cannot determine whether a user is a patient, student, applicant, or ophthalmologist, we must rely on the behavior of visitors to the website to draw conclusions. This includes the method of acquisition, primary landing page, further interactions on the website (further page visits, downloads, and requests for appointments), the time spent during the session and on the different pages of the website.
Our eye hospital's institutional website comprised a total of ≈100 pages during our evaluation (range 60–100). The website's structure started with a total of 5 levels in 2016. It was reduced to 4 levels in February of 2017 (Table 1). During this period, we strongly abbreviated the page file names. All file names had contained complete terms that were easily comprehensible, even for people not involved in the website's design and maintenance. Except for the main page, all subpages now carry highly abbreviated names. In July 2017, a rotating banner was implemented on the main page that triggered automated page reloads every 5 seconds. Figure 1 shows screenshots of the main page before and after the change.
2.2 Data collection
The access data from our institutional website was assessed with Google Analytics. Data collection started in 2016. We analyzed the overall user parameters from Table 2. The number of users and the acquisition sources were analyzed using analysis of variance (ANOVA). Statistics were done with R. We also analyzed the number of visits on our subpages and the time spent on them, as well as further interactions and the bounce rate. To draw conclusions on our target audiences, we classified the content on those pages in the different categories (patient-relevant, medical student-relevant, applicant-relevant, and professional-relevant content). We also assessed the number of requests for appointments from our online form from an internal source. Since we exclusively used anonymized data, ethical approval was not necessary.
3.1 Overall users
An overview of user numbers and behaviors is found in Table 2.
At the beginning of the time frame in 2016, we counted around 8000 users per month with 10,000 sessions. There was a drop in user numbers in April 2017 from ≈8000 users per month to ≈6000. The number of sessions decreased from approximately 10,000 to 7000 in the same timeframe (Fig. 2). The bounce rate during that time decreased from 60% to 50%. It continued to fall in July 2017 to under 1% due to changes on the home page (a rotating banner had been placed on the home page) and will therefore not undergo further analysis after that unfortunate modification.
We counted 2456 returning users in 2016 compared to 79,116 new users. Also 2017 revealed 10.319 returning users compared to 58,037 new users. In 2018, we noted 10.848 returning users compared to 56,892 new users. In 2016, 35% of sessions involved returning users, whereas in 2017, 40% of sessions involved returning users, and in 2018, 40% of sessions were also from returning users.
The most frequently visited pages were the main page with 28.2%. From there, 55.6% of users continued to at least one other page (with information for patients the one accessed most often). The 2nd most-visited page was the webpage of the above-mentioned staff member with 11.1%. However, 97% of sessions ended on that page.
Acquisition data were only available from 2017 and 2018. Most users were directed to the website by an organic search (54.4%) followed by web referral (23.1%) and direct access (22.4%). In Google Analytics, the term “organic search” is used to summarize all traffic through search engines (this does not include paid advertisements). Interestingly, over 70% of web referrals came from the web page of a particular employee at our eye hospital who used to redirect from his well-known optical illusion site to his personal page on our hospital website.[14,15] Only 0.5% of referrals were from the main university hospital website. Moreover, over 38% of direct accesses were on the same page. The 2nd most frequently visited page was our main page with a share of 35%. Google was the dominant search engine that led to our website (68,578 users), followed by Bing (5039 users) and Yahoo (621 users). The most common search term was the name of our department, followed by various disease names (most common: pterygium, corneal dystrophy).
Interestingly, the number of new users per month as well as referrals by organic search dropped after March 2017, whereas direct access stayed the same (Fig. 2). The decrease in the number of new users, organic search referrals and other referrals was statistically significant (Fig. 3, Table 3).
3.4 Utilization of the website by target audiences
The number of page views and percentages for the different types of content are summarized in Table 4. The pageviews on pages relevant for medical students revealed an increase from 2016 to 2018. The pageviews on pages relevant for patients and for ophthalmologists did not change. The pageviews on the page for applicants declined. Patient-relevant traffic is the largest, followed by that of ophthalmologists and medical students. Patient-relevant behavior was evaluated on the patient-specific page (Fig. 4). Before the change to our website structure, 14.6% of users here entered through this page. Of the 85.4% who had originally accessed a previous page of our website, 55% came from the main page. About 5% came from the page relevant for ophthalmologists. Most other accessed pages included content about therapeutic options for diseases. About 33% of users exited the website on this page. Of the 77% who visited another page, 37% went to the main page, the other subsequent pages also covered content about diseases or therapeutic options. Of those who came from a previous page of our website, 46% came from the main page, and 6% from the page relevant for ophthalmologists. Here, too, most other previous pages included content about therapeutic options for diseases. 42.7% of users exited the website on that page. Of the 57.3% who visited another page, 33% went to the main page, the remaining pages contained information on the hospital staff and organization.
We implemented an online appointment request (in April 2017) on the patient-relevant page. A total of 2985 requests had been made by the end of 2018. The online-appointment page was viewed 28,986 times during that time period. Before and after the implementation of our online appointment form, the total number of patients in our outpatient department stayed roughly the same (2016: 77,456, 2017: 76,363, 2018: 76,617).[16,17]
Behavior of referring ophthalmologists was evaluated on the ophthalmologist-specific page. Before the change to our website structure, 9% of users on this page entered through this page. Regarding users originating from a previous page of our website, 34% came from the main page and 14% came from the patient-relevant page relevant. Most other previous pages included content about our hospital's organization. About 20% of users exited the website on this page. Of all users visiting a further page, 22% went to a subpage for patient-referral, 20% went to the main page, and 14% went to the page dedicated to patients. The other subsequent pages also had content about our hospital's organization. After the structure change, 6% of users entered through this page. About 41% of users came from the main page and 22% came from the patient-relevant page. Here, too, most other previous pages provided information about our hospital. And 19% of users exited the website on this page. Of all users visiting a further page, now only 17% went to a subpage for patient referral, 20% went to a subpage about the hospital staff, and 19% went to the page dedicated to patients. The other next pages also included information on our hospital's organization.
The behavior of medical students was evaluated on the page dedicated to medical student information. Before the change to our website structure, 13% of this page's visitors entered through this page. About 56% came from the main page; most other previous pages include subpages for medical students. And 21% of visitors exited the website on this page. Of all users visiting a further page, 66% went to subpages containing further information for medical students, 7% went to the page with information about job applications. After the structure change, 19% of users on this page entered through this page. Also 50% of users came from the main page; the other previous pages included information on our hospital's organization. In addition, 61% of users continued to the subpages providing more information for medical students. Also 8% continued to information for job applications and 25% of visitors exited the website here.
Job applicant behavior was evaluated on the page dedicated to job applications. Before the change to our website structure, 11% of users on this site had entered through this page. Of the 89% who had come from a previous page of our website, 31% came from the main page, and 9% came from a subpage regarding further information about job applications. Most other previous pages included content about our hospital's organization. Also 21% of users exited the website on this page. Of the 79% who visited another page, 49% went to the subpages with further information about job application. After the structure change, 16% of that page's users had entered through the job-application page. The most frequent previous pages were the main page (30%) and the medical student-relevant page (12%). Here, too, most other previous pages contained information on our hospital's organization. Also 22% of users exited the website on this page, and 27% of users went to the subpages with further information about job applications, the other next pages now contained information on our hospital's staff and organization.
The 2018 user flow from the main page to the group-relevant pages is summarized in Figure 5.
The pages with the longest viewing time (only counting pages with more than 1000 visits) were on information for medical students (around 6 minutes) and on the page for applications (around 5 minutes).
The loading time of the website was 3.89 seconds in 2016 and 2.85 seconds in 2018. In the month before the website change, the loading time was 3.50 seconds. In the month after the change, the loading time worsened to 5.98 seconds.
Of all downloadable files on our website (data only available for 2017 and 2018), the file for patient referral was downloaded the most (27.0%), followed by the curriculum vitae of the head of department (6.31%) and the curriculum for medical students (6.25%).
The number of total users and new users was the highest in 2016, and lower thereafter. The exact timepoint of the drop in users can be sharply pinpointed to April 2017 (Fig. 2). This correlated closely with the systematic shortening of all file names on the website as well as the change in the website hierarchy 1 month earlier (Table 1). We believe that these changes may have negatively affected our Google Page Rank, because the decrease in the number of sessions is proportional to the decrease in users who were referred to our page by organic search. However, the exact mechanism of this drop in views remains unclear.
The traffic from returning users ranged from 35% to 40%, with a higher number of returning users in 2017 and 2018. Otherwise, the drop in users due to fewer referrals from organic search would have been more drastic.
The drop in users might also be caused by alternate information resources for patients on the internet. Wikipedia and social media sites are known to be used by patients. Benetoli et al reported that social media sites are used by patients for group discussions, Wikipedia is a source for factual information about diseases, and YouTube can be used to gain information on medical procedures and operations. Interestingly, the bounce rate decreased around the time of the structural changes to the website from 60% to 50%. It is difficult to interpret this bounce rate. The visitor either left the website without having interacted due to the site's irrelevance or because of having found information on the first page and not needing further interaction. The 2nd drop in the bounce rate to under 1% is most likely due to the implementation of a rotating banner. It triggers a page reload without user interaction and is thus registered as interaction. This reduces the bounce rate. Therefore, the low bounce rate since July 2017 is unsuitable for further analysis.
There are few reports on the bounce rates of websites dedicated to medical information. Gordon et al reported a bounce rate of 62% to 83% on a website dedicated to living kidney donation during a campaign. Crutzen et al reported a bounce rate of 32.9% on a website dedicated to sexual health. Brouwer et al reported a bounce rate of more than 50% on a lifestyle-intervention website. Overall, our bounce rate seems to be within the reported range; however, experience in health-related websites is currently very limited. Future approaches like Web Service Composition and Service-Oriented-Architecture might drastically change user behavior.
We had expected to observe our home page's dominance, since most users land there. Since our home page is designed to provide an overview of information as well as links to navigate on our website, the flow of the visitors to other parts of our sites is normal. The high number of last or sole visits to the third author's site means that either the visitors landed there by accident or they found the information they had been seeking.
Most of the user interest was for the patient-related parts of our website and for referring colleagues. The information for medical students and applicants was not viewed as many times; however, the mean time that users spent on those pages is far greater than that spent on the other parts of our site. This suggests that this information is indeed relevant, but for fewer people.
Jyot et al analyzed the traffic on their surgical educational website which is aimed at trainees. They reported over 6000 visits by 257 users during a time interval of 18 months and an average length of visit of 14 minutes. However, their setting is not entirely comparable, since it contains more material and videos and is aimed at trainees and not at medical students.
Our high number of referrals is caused by 1 link from an external website that leads directly to the personal page of 1 employee at our eye center. This external website is well-known and attracts a lot of traffic for the optical illusions presented on the site. Direct access is always assumed when no source is visible, either due to direct access through the browser, or during referral from a different site without identifiable information. Since we do not use the conversions for analyzing our website, a more detailed analysis is not possible at the moment.
Bardach et al reported a high rate of search engine access on 18 websites dedicated to online public reports of hospital or physician quality. In concordance with our study, the most common search terms were the name of the hospital or the website. The disease names used as search terms are diseases in which our center specializes.
4.4 Utility of the website for the target audiences
Since we cannot clearly associate any one user to one of our target audience groups (patients, referring ophthalmologists, medical students, job applicants), we can only analyze the traffic and session duration on the relevant pages and use independent data such as file downloads or the number of appointment requests.
We had expected that the patients’ group would be the most frequent among the users. Accordingly, the patient-relevant page was visited the most often among the group-relevant pages (Fig. 4, Table 4). Regarding the traffic on that page, we noted many visits and possibly returns from pages giving information about various diseases and treatment options. This behavior seems fitting for a patient, since the information on diseases is a main interest. There is traffic to and from the page relevant for ophthalmologists. This still seems plausible, since either a patient or an ophthalmologist might want to have a look on other specific pages for additional information. On the patient-relevant page is also a tool for appointment requests for our hospital. This online form was introduced in April 2017 and was not advertised. The number of online-requested appointments (2985 in 20 months) was low compared to the total number of patient visits in our center. Nevertheless, 10.3% of views on patient-related pages led to a request for an appointment in our hospital. This number is quite high considering that most of our patients are elderly. However, most of our patients are still referred by ophthalmologists via fax. The total number of patients in our hospital remained unchanged throughout the introduction of our online appointment request with over 76,000 patient visits per year. In regard to page views, the implementation of our request tool did not lead to an increase in page views on that page. However, we feel that this tool is important to ensure access to our hospital, even when no referring ophthalmologist is available close to the patient. Also, this method of data transfer is safer in terms of data privacy protection than is data transfer via fax.
The page for referring ophthalmologists was the 2nd most visited 1 among the group-relevant pages. This is surprising, since the number of local ophthalmologists is lower than the number of medical students in our institution during 1 year of medical school. However, the behavior with visits to sites mainly about our hospital's organization and especially the traffic to the patient-referral page seem suitable for this target audience. Also, note the still-high number of downloads of means of referring a patient to our institution.
Pages for medical students and job applicants attracted the least traffic (with a slightly higher number for medical students). Medical students reveal high traffic to and from pages containing further information about timetables and other student-related content. From the main students’ page, some users go to the information on job application, which is not surprising, since students are potential job applicants. On the site for job applicants, there is traffic to and from pages with information about our hospital's organization and pages about further relevant information about applying for work at our institution. Both are equally important for a user interested in a job at our hospital. The pageview durations differ among the 4 target audiences. The pages for medical students and job applicants are viewed far longer (6 minutes; 5 minutes) than are the other pages of our website. We assume that due to the high amount of information on those pages, targeted users stay longer to obtain everything they find relevant. The overall reduction of leading time contributes to a more convenient experience on our website. The drastically longer loading time directly after the website change in February 2017 might have also harmed the user numbers.
One of the major limitations of our study is its retrospective nature. We did not optimize our website for data acquisition via Google Analytics and have only a limited subset of the full potential at hand. That is, we did not configure conversion endpoints for designated targets on our website. Moreover, Google Analytics is usually used for marketing purposes, and not for analyzing health institution websites. We plan to add more parameters (i.e., conversions) for a more detailed analysis of user behavior in the future.
We herein present website usage data on one of the largest German university eye hospitals over 3 years. We demonstrate that changing the structure of website (layout, hierarchy, and file names) can unexpectedly reduce the number of users as well as the interpretability of the usage data. Therefore, such changes should be done cautiously and be closely monitored. In summary, our website is used by all our target audiences. The behavior of and number of users from each group differ. To our knowledge, data concerning the use of Google Analytics combined with in house data to estimate the utility of the website of a large tertiary referral center is very limited. However, the method seems to be applicable for a greater variety of medical institutions’ websites.
Conceptualization: Stefan J. Lang, Daniel Böhringer, Michael Bach, Thomas Reinhard.
Data curation: Stefan J. Lang, Daniel Böhringer.
Supervision: Michael Bach, Thomas Reinhard.
Writing – original draft: Stefan J. Lang.
Writing – review & editing: Daniel Böhringer, Michael Bach, Thomas Reinhard.
. Druce I, Williams C, Baggoo C, et al. A comparison of patient and healthcare professional views when assessing quality of information on pituitary adenoma available on the internet. Endocr Pract 2017;23:1217–22.
. Zwier S. “On the doctor's orders”: a pilot study of the effects of website
marketing for medical specialist providers under gatekeeping arrangements. Health Mark Q 2017;34:233–46.
. Kern C, Fu DJ, Kortuem K, et al. Implementation of a cloud-based referral platform in ophthalmology: making telemedicine services a reality in eye care. Br J Ophthalmol 2019;104:312–7.
. Muzammal M, Talat R, Sodhro AH, et al. A multi-sensor data fusion enabled ensemble approach for medical data from body sensor networks. Information Fusion 2020;53:155–64.
. Pirbhulal S, Wu W, Li G, et al. Medical information security for wearable body sensor networks in smart healthcare. IEEE Consum Electron Magazine 2019;8:37–41.
. Wu W, Pirbhulal S, Zhang H, et al. Quantitative assessment for self-tracking of acute stress based on triangulation principle in a wearable sensor system. IEEE J Biomed Health Inform 2019;23:703–13.
. Huerta TR, Hefner JL, Ford EW, et al. Hospital website
rankings in the United States: expanding benchmarks and standards for effective consumer engagement. J Med Internet Res 2014;16:e64.
. Huerta TR, Walker DM, Ford EW. An evaluation and ranking of children's hospital websites in the United States. J Med Internet Res 2016;18:e228.
. Raban MZ, Tariq A, Richardson L, et al. Evaluation of web-based consumer medication information: content and usability of 4 Australian websites. Interact J Med Res 2016;5:e21.
. Pang PCI, Chang S, Verspoor K, et al. Designing health websites based on users’ web-based information-seeking behaviors: a mixed-method observational study. J Med Internet Res 2016;18:e145.
. Himmel W, Meyer J, Kochen MM, et al. Information needs and visitors’ experience of an Internet expert forum on infertility. J Med Internet Res 2005;7:e20.
. Tian H, Brimmer DJ, Lin JM, et al. Web usage data as a means of evaluating public health messaging and outreach. J Med Internet Res 2009;11:e52.
. Suzuki T, Tani Y, Ogasawara K. Behavioral analysis of visitors to a medical institution's website
using Markov chain Monte Carlo methods. J Med Internet Res 2016;18:e199.
. Bach M. 135 Visual Phenomena & Optical Illusions [Internet]. [cited August 3, 2019]. Available at: https://michaelbach.de/ot/
. Accessed August 3, 2019.
. Bach M. Visual illusions on the Internet: 15 years of change in technology and user behavior. Perception 2014;43:873–80.
. Schoelles KJ, Ludwig F, Lang SJ, et al. Advanced training curriculum at the eye center, University of Freiburg. Klin Monbl Augenheilkd 2020;237:310–8.
. Sinha MS, Freifeld CC, Brownstein JS, et al. Social media impact of the food and drug administration's drug safety communication messaging about zolpidem: mixed-methods analysis. JMIR Public Health Surveill 2018;4:e1.
. Benetoli A, Chen TF, Aslani P. Consumer health-related activities on social media: exploratory study. J Med Internet Res 2017;19:e352.
. Gordon EJ, Shand J, Black A. Google analytics of a pilot mass and social media campaign targeting Hispanics about living kidney donation. Internet Interv 2016;6:40–9.
. Crutzen R, Roosjen JL, Poelman J. Using Google Analytics as a process evaluation method for Internet-delivered interventions: an example on sexual health. Health Promot Int 2013;28:36–42.
. Brouwer W, Oenema A, Raat H, et al. Characteristics of visitors and revisitors to an Internet-delivered computer-tailored lifestyle intervention implemented for use by the general public. Health Educ Res 2010;25:585–95.
. Sun X, Wang S, Xia Y, et al. Predictive-trend-aware composition of web services with time-varying quality-of-service. IEEE Access 2020;8:1910–21.
. Jyot A, Baloul MS, Finnesgard EJ, et al. Surgery website
as a 24/7 adjunct to a surgical curriculum. J Surg Educ 2018;75:811–9.
. Bardach NS, Hibbard JH, Greaves F, et al. Sources of traffic and visitors’ preferences regarding online public reports of quality: web analytics and online survey results. J Med Internet Res 2015;17:e102.