Improving Web Site Performance Using Commercially Available Analytical Tools

Abstract

Background 

It is easy to accurately measure web site usage and to quantify key parameters such as page views, site visits, and more complex variables using commercially available tools that analyze web site log files and search engine use. This information can be used strategically to guide the design or redesign of a web site (templates, look-and-feel, and navigation infrastructure) to improve overall usability. The data can also be used tactically to assess the popularity and use of new pages and modules that are added and to rectify problems that surface.

Questions/purposes 

This paper describes software tools used to: (1) inventory search terms that lead to available content; (2) propose synonyms for commonly used search terms; (3) evaluate the effectiveness of calls to action; (4) conduct path analyses to targeted content.

Methods 

The American Academy of Orthopaedic Surgeons (AAOS) uses SurfRay's Behavior Tracking software (Santa Clara CA, USA, and Copenhagen, Denmark) to capture and archive the search terms that have been entered into the site's Google Mini search engine. The AAOS also uses Unica's NetInsight program to analyze its web site log files.

Results 

These tools provide the AAOS with information that quantifies how well its web sites are operating and insights for making improvements to them.

Conclusions 

Although it is easy to quantify many aspects of an association's web presence, it also takes human involvement to analyze the results and then recommend changes. Without a dedicated resource to do this, the work often is accomplished only sporadically and on an ad hoc basis.

Introduction

When content is published in a newspaper or broadcast on radio or television, it is difficult to directly measure its impact (number of readers, listeners, and viewers) and the available indirect methods (circulation statistics, polls, surveys, and so on) can be costly and perhaps not timely [⁶]. The data will be at best an estimation or approximation of the actual activity. Measuring the success of calls to action (buy this product, write to Congress, and so on) is also not straightforward. In contrast, the use, impact, and success of electronic publishing (for example, through web sites and e-mail messages) can be directly measured because it is possible to log every activity and then analyze, consolidate, and summarize the resulting information in timeframes that approach real-time. This allows electronic publishers to rapidly assess the efficacy of their web designs (template, navigation, and look-and-feel), marketing strategies, and the popularity of their content and to then quickly make modifications to improve overall use and usability [^{1, 6}].

In September 2006, the official web site of the American Academy of Orthopaedic Surgeons (www.aaos.org) was relaunched after the staff carefully designed and executed usability and card sort tests [⁵]. (These tests were conducted by an external web consulting firm and a major university.) The old site's structure had degenerated into a patchwork framework of disjointed modules, some of which dated back to 1995 when the site was first launched. The new site now presents content as users (AAOS members, allied healthcare professionals, patients, and the general public) want to see it and eliminates the old layout that in reality reflected the association's internal organization chart. At the same time, the AAOS purchased several commercially available analytic tools to allow their programming staff to accurately measure web site traffic and to quickly detect and rectify issues as they surfaced. These included Unica's NetInsight web log file analysis system [Unica Corporation, Waltham, MA; http://www.unica.com/products/enterprise-web-analytics.htm] and the Mondosoft [now SurfRay, Santa Clara, CA and Copenhagen, Denmark; http://www.surfray.com] Behavior Tracking module. The latter product can be used in conjunction with the previously installed Google Mini search engine [Google Inc.; http://www.google.com/enterprise/search/gsa.html].

Software tools described in this paper have been used to: (1) Inventory search terms that lead to available content. This indicates what information users are looking for and whether they were successful in finding it. (2) Propose synonyms for commonly used search terms. Often, the vocabulary used by the casual searcher does not match technical terminology contained in the content. Using alternative terms and spelling can make content more visible. (3) Evaluate the effectiveness of calls to action. These include items such as product purchases and political action committee (PAC) donations. (4) Conduct path analyses to targeted content. Web designers usually think they know how viewers reach internal sections of a web site, but path analysis can identify all of the various paths used by viewers to reach designated pages.

Web Analytical Tools

The most universally known and quoted web analytics are page views and visits over time [¹], but web analytics packages such as NetInsight can perform much more sophisticated analyses such as: (1) path analysis: how web users traverse a site; (2) assessment of content popularity: for example, ranking by type the most frequently viewed articles in a journal or news magazine; and (3) completion rates: the percentage of users who purchased a product after placing it into a shopping cart or the number of members who sent a message to Congress after reading a position paper.

NetInsight

Web site servers, if configured appropriately, log information concerning each page that is sent to an end-user's browser. This information contains data such as the time and date of the page request, the viewer's web browser brand and operating system, the preceding page (termed the referrer) that was displayed, and many other pieces of information. Programs such as NetInsight ingest this log data (which can be gigabytes in length) and then, through standard and custom reports, determine how many people used a web site in any given time, which pages were viewed, how many pages on average were requested by each visitor, where that visitor was located, and how he reached the target web site or discrete web page.

Behavior Tracking

Behavior Tracking provides robust statistics, information, and insight into how well the association's search engine is performing as well as guidance into how to improve the overall search experience and the underlying web content. Commonly tracked parameters include: (1) search terms for which content was found; (2) search terms for which content was not found; (3) the percentage of search results pages for which at least one result was chosen (clicked); and (4) the various search parameters that were used to retrieve any specific page.

Google Mini

The Google Mini is a search appliance (a self-contained computer or server) that resides on a company's network that indexes the content on its web servers. It is similar in concept to the large search engines commonly used on the Internet (Google, Yahoo, Bing, etc.). Among other capabilities, the Google Mini can be programmed to accept synonyms for words or phrases and to prioritize the search results list if specified key words have been entered.

Important Features to Track

Trends in Search Terms

One aspect of Behavior Tracking is to quantify over time the usage of various search terms. One would expect that the frequency of occurrence of “orthopaedic surgeon” or “orthopaedics” would remain constant throughout the year while terms like “Annual Meeting” or “lawn mower safety” may rise just before registration for the annual meeting opens or as spring arrives.

Shopping Cart Abandonment

Shopping cart abandonment is an important parameter to monitor for any web site that sells products. There are many factors that affect this statistic, which is generally defined as the percentage of customers (in a specified timeframe) who do not eventually purchase something compared with those who placed at least one item in an electronic shopping cart. That is, the abandonment rate equals, for any given timeframe: (1 − total purchases made/total shopping cart sessions) * 100%. A shopping cart session is defined as an instance where one or more items are placed in the electronic cart.

Path Analysis

Another important capability of log file analysis programs is path analysis. This makes it possible to determine and quantify the steps a web visitor takes from any initial starting page, the actions that were taken that eventually led to any designated target page, or the pages that were viewed between a given starting and ending page. These measurements can provide significant insight into the overall usability of a large web site [⁶]. For example, if the web design's intent is to quickly lead the user from a landing page to a donation form (perhaps for orthopaedic research or the political action committee), it is easy to verify that the majority of contributors took the intended path and did not meander throughout the site either because of navigation issues or an overall lack of interest. Care must also be taken to ensure that web pages laden with graphics, audio clips, or videos do not substantially increase the overall loading and rendition times. Visitors are not likely to tolerate multimedia effects that delay finding what they are ultimately looking for. NetInsight contains features that can monitor any specific page's average time to load as well as the number of times that a page is abandoned prior to total rendition (defined as an aborted request). These statistics can be used to identify slow loading pages and those with high abort rates. (The AAOS has only recently begun placing videos on its site that start automatically and has not conducted any research into optimization using load time and aborted request statistics.)

Tracking at the AAOS

Designing and maintaining a complex medical web site that is used by a diverse audience (orthopaedic surgeon members, allied health professionals, patients, the press, and the general public) that is simultaneously easy to navigate and use is not an easy task. The AAOS has a broad spectrum of programs and products centered on education, advocacy, research, and communications. All of these areas have representation in the 30,000+ pages that currently comprise its web space.

This has been verified by the Behavior Tracking module's analysis of the local Google Mini's usage patterns. First, the number of new search terms used per user session is plotted against the frequency of occurrence of the top 100 terms compared with all terms used during a given timeframe. For comparison purposes, Behavior Tracking overlays similar data from approximately 100 of SurfRay's other clients and then segments the results into quadrants. For AAOS, an average of 1.11 new terms per search session is introduced, whereas the top 100 terms are only used 16% of the time. Behavior Tracking classifies the AAOS site as a portal in which users are looking for a wide variety of content that is not strongly correlated and where each user tends to introduce new search terms (Fig. 1). (The other quadrant classes are Company Branded [all users tend to be looking for the same, relatively small number of things. This might typically mean that they expect the site to describe a specific company, brand or strong product line.], Focused Information Provider [users expect to find a somewhat wide, but still sharply limited set of information. This type of site has substantially more than 100 search terms that are most popular (eg, 300 words), but users rarely submit words beyond this list.], and Vertical Portal [users expect comprehensive information about a single subject or a small group of similar subjects. They favor the most popular topics, but also show a diverse interest related to those topics.]). Optimizing search and navigation on a portal is very difficult because of the extremely varied content and the diversity of users. For portals, it is important to constantly monitor the lists of top search terms and top not-found terms to (1) provide content that the viewer is seeking; and (2) develop synonyms and alternate spellings for items on the not-found list that actually have relevant web pages associated with them. (AAOS statistics are for the 2-week period ending August 13, 2009.)

Fig. 1: Behavior Tracking maps the AAOS coordinates (1.11, 16%) onto a graph termed an Expectation Map that is divided into quadrants.

For example, the AAOS has a noteworthy patient education web site (Your Orthopaedic Connection; www.OrthoInfo.aaos.org) that is used extensively by the public. Care must be taken to ensure that content can be located with verbiage that laymen will use in lieu of precise medical terminology that either is not part of their vocabulary or where the correct spelling is not known [^{2, 6}]. Illustrations include plantar fasciitis, carpal tunnel, and meniscus. When AAOS first implemented Behavior Tracking, common misspellings for all of these terms appeared on the Top Not-Found Terms list, which implied that, although content existed on the web site, users could not locate it unless they knew in advance the correct spelling and terminology. Armed with this information, it was easy for the programming staff to insert synonyms into the Google Mini. For instance, the search engine now displays, for misspellings such as planter fasciitis, plantar fascitis, and planter fascitis, the message “Did you mean? plantar fasciitis.” Similarly, marketing terms can be confusing, even to those who comprise the target audience. AAOS sells a series of coding books under the title Orthopaedic CodeX (a coding and cross-reference tool; http://www4.aaos.org/product/productpage.cfm?code=05124). However, the membership often thinks Code X or Code-X. Once Behavior Tracking highlighted this issue, it was easy to resolve.

By itself, the Google Mini has the flexibility to interpret designated keyword parameters as high priority and then force the corresponding content to the top of any returned search results list. This is referred to as a key match entry. In 2007, the AAOS launched its new Orthopaedic Surgeons Disclosure Program and a significant amount of publicity for this important new project was also prepared. To ensure that the membership could easily find the correct web pages, a key match entry was created in the Google Mini that displays “Key Match - Orthopaedic Surgeons Disclosure Program” as the first entry in the search results list whenever any term that includes disclosure is entered. Without using this feature, a simple query using disclosure as the search parameter will currently return over 880 links spread over almost 90 pages. This would force the user to either refine his or her search or wade through the various possibilities. Research suggests searchers invariably will review only the first one or two search results pages [⁷].

Behavior Tracking also automatically tracks trends in search term use over time. One standard comparison calculates, for the top 100 terms used during any designated time period, the frequency of occurrence of each of those terms compared with the preceding time period. As an example, 2010 annual meeting ranked 18th as of August 16 compared with 71st for the period ending August 2. This is expected because housing for the 2010 AAOS Annual Meeting opened on August 11 and members are always anxious to reserve hotel rooms far in advance of the meeting. If use of this term had not increased, it might indicate that members were having significant problems finding Annual Meeting information (and specifically housing) on www.aaos.org, particularly if hotel registrations were not strong (compared to previous years in previous cities). Alternatively, it might indicate that the web site's navigation was working properly and that users had no need to conduct a search. The key is that this type of parameter can be used as an early warning indicator of potential problems.

Lastly, AAOS has occasionally leveraged another module that is included in the Behavior Tracking program, BehaviorMatch (SurfRay; http://www.surfray.com) (Table 1). The software tracks each search session and makes inferences as the user refines searches that are not successful (either because they return no results or none of the displayed links are clicked). For example, there are over 660 result pages with the term screw but only about 200 for pedicle screw. Instead of forcing the user to refine his or her search (or risk his or her abandoning the site altogether), the Google Mini search engine could be programmed to make screw a synonym of pedicle screw or declare it a key match. Some of the currently suggested synonyms include acl (acl reconstruction, acl tear, mcl, meniscus, pcl), carpal tunnel (arthritis, carpal tunnel exercise, carpal tunnel syndrome, trigger finger), and frozen shoulder (adhesive capsulitis).

Table 1: BehaviorMatch suggests synonyms and alternate phrases that web users may use as search terms

Just as the Behavior Tracking software helps to optimize the search experience on the AAOS web properties, the Unica NetInsight web analytics program allows programmers and marketing staff to measure use of any section or page on the site and to assess the success of various campaigns (such as product purchases, PAC donations, and so on). The available statistics, trends, and filters are far more extensive than the elementary page view and visit statistics that often are quoted in popular literature and company press releases.

Custom NetInsight reports indicate that in 2006, the AAOS shopping cart abandonment rate was approximately 17.4%; in 2009, this had decreased to 15.4% for product purchases (Fig. 2). These rates are low compared with industry norms that usually exceed 50%, but it is expected that medical specialty associations, whose members are quite homogenous and whose products are fairly specialized, will have an abandonment rate below commercial sites that cater to the entire population and have a wide variety of offerings.

Fig. 2: In 2009, there were 10,929 sessions where the electronic shopping cart was loaded with one or more item. Of these sessions, 10,497 reached the “Confirm Order” stage and, of these, 9,249 resulted in purchases.

In August 2009, the AAOS placed an animated graphic on the homepage of www.aaos.org that featured the upcoming 2010 Annual Meeting. Clicking the graphic started a short video that featured key AAOS members who described the meeting site (New Orleans) and the opening of hotel registration. The video ended with an option to “click here” on the movie screen to go to the hotel registration web page. Several days after posting, we performed a path analysis to determine if viewers were transferring to the designated page. Approximately 64% (100 of 156) viewers went directly to this page; most of the others first navigated to the Physician Education landing page before finding the hotel registration link. Subsequently, the design was modified to display text links (on the pop-up that embeds the video) that explicitly guide the viewer to the hotel registration page. Some users probably did not realize that they could simply click the movie screen. Alternatively, others may have an aversion to multimedia pop-ups.

When it appeared that AAOS might not reach the hotel room reservation numbers that were guaranteed in its contracts, an embedded, auto-play movie was placed on the www.aaos.org homepage that featured several members who described the upcoming Annual Meeting and the various venues associated with it. This increased hotel reservations by several hundred in the last weeks preceding the meeting and the guarantees were met.

The AAOS Annual Meeting registration system also includes several helper programs that provide guidance on course selection (“CME Wizard”) and that create personal schedules or calendars for attendees (“Make Your Own Schedule”). Shortly after registration for the 2010 meeting began, staff noticed that the calendar function was not being used nearly as frequently as in past years. A quick path analysis showed that users were having difficulty finding the module in the Annual Meeting web site. After it was moved to a more logical section of the web site, usage improved.

Recently, the AAOS purchased and installed a Lyris list server [Lyris Inc.; http://www.lyris.com] that will eventually replace another product that has been in use since the late 1990s. Lyris has the intrinsic capability of measuring e-mail open rates and click-through rates (from embedded links within the message). These data will be useful for prospective advertisers who may want to place ads within the association's electronic newsletters and simply to measure the overall popularity of the many messages that are sent on a weekly basis to AAOS members. The information can also provide helpful information concerning the effectiveness of placing embedded links in various designated areas. However, e-mail open rates are typically measured by embedding into the message a one-pixel graphic that must be retrieved (and therefore logged) by the sending server. These open rates can be artificially lowered by e-mail clients that, by default, prevent loading external images. And they can be inflated by programs, such as Microsoft Outlook, that may have preview panes that load the graphic regardless of whether the user actually opens and reads the message or simply deletes it. While it may not be possible to determine with 100% accuracy how many messages are actually opened, it is possible to assess, over time, whether the trend (for items such as electronic newsletters) is increasing, decreasing, or remaining stable.

Discussion

Reports that track the popularity of all published articles over time are useful tools for journal and news magazine staff, editors, and publishers. These provide valuable insight into where the membership's interest lies (at least with respect to the current portfolio of topics) and the overall lifespan of those articles or topics. Additional articles can be planned for highly read examples that have no counterparts and continued emphasis can be given to those topics that are always viewed. Infrequently read articles may indicate an overall lack of interest in a specific topic (and therefore it should be dropped from future consideration) or may mean that it needs to be presented in a different manner. For example, the most frequently viewed AAOS Now (the monthly AAOS news magazine) articles are coding topics followed by descriptions of new or revised clinical procedures. This type of information is regularly reviewed by the editorial staff that determines future content.

This paper has outlined how analytical tools can be used to analyze and improve web site usage and performance by capturing and reviewing basic parameters. These include: (1) Search terms that lead to successful searches and, perhaps more importantly, unsuccessful searches. The latter can be used to assess whether users cannot find existing material and to suggest future content for material that does not currently exist. (2) Synonyms and alternate terminology for accessing existing material. Medical sites often use a complex vocabulary that the general public may not understand. It is important to map, in a web site's search engine, lay terms that correspond to their medical counterparts. Search term misspellings should also be anticipated and accommodated. (3) Effectiveness of calls to action. Interactive web sites can often be used to persuade a viewer to buy a product, make a donation, lobby Congress, etc. While persuasive content is always needed to initiate any of these actions, the completion of them is often dependent on a clear, straightforward system that quickly gathers the required user information and, if applicable, payment data. Analyzing the drop-off rates from step-to-step in this process can often provide valuable insights into improving and simplifying the process. (4) Path analysis. Large web sites can quickly become nightmares to navigate. Web designers often believe that users will quickly go from Point A (perhaps the homepage) to Point B (a landing page) to Point C (the target content). In actuality, there are usually many ways for users to find the desired end point and just as many ways for them to get lost. A detailed analysis can delineate all of the paths people use to reach an end point or the paths they take between two designated pages. This information can then be used to simplify the navigation, nomenclature, and link labeling if problems are detected.

The literature contains many references (some cited in this paper) that describe the fundamentals of web site usability and redesign as well as the use of log file analysis programs that create numerous parameters to assess the overall effectiveness of a web site. To a lesser extent, the capture and analysis of data entered into search engines is covered in the literature. This paper documents how the data generated by these analytical tools has been used to evaluate the performance of AAOS' web sites and has then guided some of the design improvements that have been made over the past several years.

Determining why customers abandon in-process purchases can help lower the overall rate. In general, as the process is simplified, the conversion rate should improve. When AAOS first began using electronic commerce, the actual purchase process (identifying the customer, obtaining shipping and billing information, and capturing a credit card) required many web screens and the user was never told how many total steps would be needed to complete the transaction. AAOS now clearly indicates on its web pages how many steps are needed to make a purchase and the step that the customer is currently at. At present, only four steps, including the confirmation page that displays a receipt, are required. Some studies suggest shopping cart abandonment can be further minimized by listing, as soon as possible, shipping charges and taxes [²]. This is something that AAOS currently only does when the screen that requests payment is displayed. The next generation of AAOS' association management system may allow these charges to be displayed as items are added to the cart.

The AAOS also uses its web site to publicize committee and council vacancies and to then accept applications for these openings. Termed the Committee Appointment Program (CAP), this module initially used a complex system of screens and forms to capture the information that is used by the Committee Appointment Program Committee to review, vet, and appoint new committee members. The program frustrated many applicants and AAOS staff fielded many phone calls that requested help with the application process. The program was then streamlined and simplified and, perhaps most importantly, the number of steps needed to complete the application was prominently displayed on each screen. Almost immediately, the number of support calls to staff diminished.

This technique can also be used to measure the success rates for call to actions. For example, AAOS places, within the product order confirmation page (after the purchase has been made), a solicitation for contributing to the Orthopaedic PAC. This was done to increase PAC donations and to provide a convenient method for reminding members of its advocacy program. This option, however, has not been successful with a conversion rate, for 2009, of only 0.3%. Possible improvements might include replacing or modifying the existing text link with a familiar graphic or a short tagline.

The tools used by AAOS (NetInsight and Behavior Tracking) are not the only options available to those who want to collect and analyze web analytics. There are many alternatives to choose from including Google Analytics (Google Inc.; http://www.google.com/analytics) which is available free of charge. It should also be noted that Behavior Tracking, although supported by SurfRay, is no longer actively marketed by that company.

Although software tools such as Behavior Tracking and NetInsight provide extensive analysis capabilities into a web site's performance, human involvement is required to develop the pertinent queries, analyze the results, and then make the requisite changes to the web templates and navigation structure. Although the tools are in place at AAOS, there is no designated web analytics point person who has overall responsibility for these activities. Instead, this is performed on an ad hoc basis by interested staff, both by Information Services and by others in the various departments.

Usability and optimization do not just happen. Organizations that want to constantly improve their web presence must dedicate staff time to use the various tools that they have purchased. The amount of data stored in web server and search engine log files can be overwhelming, particularly for popular sites. Simply having data in this format does not mean that there will be time to regularly analyze the majority of it. Metrics derived from this information should be used to verify that an association's web site is meeting a predetermined strategy rather than just creating reports simply because the capability exists [^{1, 4, 6}]. All too often, organizations underestimate the amount of resources needed to adequately determine if their web sites meet their strategic goals and then have no mechanisms in place to take corrective action if needed [³].

Acknowledgments

I thank Tewahedo Seyoum, Lead Internet Application Developer, and Laura Stanton, Manager of Software Development and Support, for reviewing this article for accuracy and completeness. Both are with the Information Services Department of the American Academy of Orthopaedic Surgeons.