Habel, Melissa A. MPH*; Hood, Julia MPH*; Desai, Sheila MPH*; Kachur, Rachel MPH*; Buhi, Eric R. PHD†; Liddon, Nicole PHD*
The Internet is one of the most commonly accessed resources for health information.1,2 In 2008, 150 million US adults reported ever going online to look for health or medical information, amounting to approximately 81% of the adult population.3 Of adult Internet users, specifically, 84% have sought health information online.3 By offering privacy, anonymity, and convenience, the Internet provides information without risking the awkwardness many individuals anticipate or experience during healthcare visits, thus serving as a valuable information resource for sensitive matters, such as sexual health inquiries.4
Young people, in particular, benefit from the availability of sexual health information online. Adolescents and young adults aged 15 to 24 years have the highest rates for the most common sexually transmitted diseases (STDs) and are estimated to acquire nearly half of all new STDs.5 Simultaneously, this group is the largest consumer of new media,6 and more than one-quarter of young adults report that they have sought sexual health information online.7 Online informational resources provide a means to connect at-risk populations to STD prevention, testing, and treatment services, and therefore represent a strategy to reduce STD transmission.
Obtaining relevant and accurate health information online can be challenging. Between 20% and 25% of online health information seekers report feeling overwhelmed or frustrated or confused by a lack of information, or an inability to find information online.8 For many, one way to sift through massive amounts of information is through Internet search engines; two-thirds of online health information seekers begin their searches by using search engines (e.g., Google, Yahoo).6,9 Such reliance on Google and other search engines is particularly important given the manner in which searching takes place and how results are placed within standard search output. For instance, in an eye-tracking experiment of college students' searches on Google, researchers found that decisions to follow search result links were strongly biased toward links higher in position—even if the abstracts themselves were less relevant to the search.10 Other researchers have found that, even though many search engine users are able to locate accurate health information, in general, many are unable to locate local health information or resources, in particular.11
Recognizing that most quests for online health information begin with search engines (such as Google, the most commonly used search engine12,13), the purpose of this study was to provide a preliminary and explorative look at the ease of online access to geographically relevant STD/HIV testing services through a standard Google search.
Guided by previous research on the way young people search sexual health topics,11 Internet queries were conducted to locate testing services on the following topics: STDs, HIV, AIDS, Chlamydia, and Herpes. Additional queries were performed to test search results for “free STD tests” and “anonymous HIV test.” To measure how Internet resources vary in various geographic locations, the health topics were paired with 6 cities/towns in 3 US regions—northeast, south, west. In an attempt to pair large and small city in the same geographic and region of the country, we selected a convenience sample of 3 large cities (Atlanta, GA; Los Angeles, CA; and Washington, DC) and 3 small-to-medium-sized towns (Dothan, AL; Burlington, VT; and Las Cruces, NM) on the basis of researcher personal knowledge and US Census' American Community Survey. The locations differed in terms of size, racial composition, and education levels. With the exception of Burlington, VT, all locations had infection rates of Chlamydia above the national average.14
Queries were entered into the Google search engine using 3 different formats (question, phrase, and key word) for each combination of city and topic, herein referred to as a “health topic family” (Table 1). For example, to locate STD testing services in Atlanta, the following 3 queries were performed: “Where can I get an STD test Atlanta?” (question), “Get STD test Atlanta” (phrase), and “STD test Atlanta” (key word). We applied these 3 search strategies in an effort to mimic the variety of search methods Internet users might use when searching online.
Data were collected for 2 weeks in December 2008; however, all websites for each city were collected on the same day to minimize variation within city categories (e.g., all data for Atlanta were collected on a Monday). Only the first 3 web results were included in the sample, as previous research indicates that users select the first link on the results page 70% of the time15, 1 of the first 3 results 79% of the time,11 and go beyond the fifth link less than 20% of the time before trying a new search term.12 In short, most users do not go beyond the first page of search result output.16 All commercially sponsored links and Google Map results were excluded from the sample.
A coding instrument was developed to capture the following website characteristics: (1) relevancy to the search topic, (2) domain and purpose, (3) search engine result rank order, and (4) website informational content. Because multiple search strategies were used, many websites appeared more than once. If a website resulted more than once within a health topic family, it was only coded once; if a website appeared under different health topics, the website was coded for each health topic. For example, if the same website resulted from the search terms “STD test Atlanta” and “Get STD test Atlanta,” the website was included in the sample only once. However, if the same website resulted from the search terms “STD test Atlanta” and “HIV test Atlanta” the website was coded twice and assigned different unique identifiers.
The instrument was pretested by 2 trained coders with 9 web pages and revised until consensus between the coders was reached. To assess interrater reliability, the coders independently assessed a random subset of websites, accounting for 13.0% of the total sample of unique websites. Resulting Cohen's Kappa ranged from 0.61 (lowest) to 1.0 (highest), which is considered acceptable.17 Any discrepancies between coders, however, were rectified before coding the remaining websites.
Our primary measure, “relevancy,” was measured by whether a website provided sufficient information to answer the query topic. That is, did websites provide relevant answers about the specific STD/HIV testing service within a community? To be coded “yes” for this variable, a website needed to explicitly provide information on testing resources or facilities. If coded “yes,” the website was further examined for more specific questions regarding contact information, hours of operation, information on cost/payment options, and the availability of the website in an alternative language.
Basic website characteristics were also recorded, including website domain type (i.e., .com, .edu, .org, .gov, or .net), rank order in Google result output (first, second, or third), and site purpose. After reviewing an extensive number of websites, the following 10 categories were developed to classify website purpose: (1) “for-profit” business, private clinic, or lab; (2) public clinics or hospitals or health departments; (3) nonprofit organizations; (4) general information resources (Websites categorized as “general information” resources were those that shared the .com domain and provided basic information about the health topic queried); (5) online directory (Websites categorized as “online directory” were those that included a text box where users could enter search queries; for example, “City Search” and “Yellow Pages.” The information would change with each query and the search engine would generate its own terms, not necessarily the ones we used); 6) resource landing pages (Websites categorized as “resource landings” were those that provided a static list of local health services and their contact information); (7) message boards or blogs; (8) news/journal article; (9) dead link; and (10) others. On the basis of data from the US Census Bureau, cities were dichotomized into large (500,000+) or medium-small (<100,000) population size.
Data recorded on the coding instrument for each website were entered into SPSS 17.0 (Chicago, IL). Simple frequencies were performed for basic univariate descriptive purposes. For websites providing information on local testing resources, frequencies provided basic descriptive information. Supplemental qualitative data were also collected on these variables. To test bivariate associations with website relevancy, we performed Pearson's χ2 tests for the following variables: health topic, website domain type, website purpose, city location and size, and website rank order in results. Odds ratios (ORs) and 95% confidence intervals (CI) were calculated for topic, domain, and city size. Bivariate associations were investigated at both the search query level and individual search level.
Searches produced a total of 378 websites across all queried topics, of which 96 were unique. Unless stated otherwise, this research will report findings from the total sample (N = 376) (2 websites were missing because of computer error). No statistical differences were found between question, phrase, and key word search strategies. Websites hosted at .com (57.3%), an Internet domain name for corporate or business websites and .org (25.7%), an Internet domain name traditionally reserved for nonprofit websites, domains were retrieved most frequently (Table 2). In terms of website purpose, the most commonly retrieved websites were online directories (20.4%), nonprofit websites (18.3%), news articles (15.6%), and resource landing pages (15.1%). Government-sponsored sites, which comprised of public health departments, public clinics, and hospitals, only accounted for 10.8% of the sample and rarely (8.0%) appeared as the first search engine result (Table 2).
Approximately half of all websites (49.5%) and 44.9% of the unique websites provided sufficient information to find local testing resources and services for the specified sexual health topic in the specified location; however, approximately three-quarters (77.0%) of all queries yielded at least one relevant website within the first 3 results. Fewer than half (47.1%) of the websites providing relevant information provided cost or payment information; however, of those, 67.0% mentioned free STD/HIV testing or free services. Among websites that provided sufficient information to answer the query, a majority of them provided basic contact information such as a telephone number (93.2%) or an address (80.6%). The following items were included less often: maps (25.7%), directions (25.1%), and e-mail address (16.2%). Only 17.3% of the websites providing relevant information offered services or information in a language other than English; when information in another language was offered, it was most often offered in Spanish. Fewer than half of relevant websites (45.0%) listed hours of operation, and of those listing hours, 26.5% listed hours for services on the weekends and 73.5% listed outside standard work or school hours.
Table 3 illustrates the number of relevant websites by health topic and the number of relevant sites within each search query. Of note, queries by means of general search terms yielded better results than queries through specific search terms. For instance, websites resulting from STD test queries were more likely to be relevant than those resulting from Herpes queries (OR = 17.2; 95% CI = 6.6, 44.7) and Chlamydia queries (OR = 11.7; 95% CI: 4.7, 29.2). About one-quarter of queries on “Chlamydia test” (24.1%) and one-third on “Herpes test” (37.9%) yielded no relevant websites in the first 3 results, whereas searches on “STD test” always yielded at least 1 relevant website. Information on free STD testing was also not readily available; one-third of queries yielded no relevant information related to free STD testing.
Table 4 shows the relevancy of website varied by location. Search queries for Los Angeles provided the most relevant results, with 84.1% of the resulting websites providing relevant information. All search queries for Los Angeles produced at least 1 relevant result. Search results yielded relevant websites 61.9% of the time for Atlanta, 58.7% for Washington, DC, 47.6% for Burlington, and 29.5% for Las Cruces. Results were least relevant for Dothan: only 15.9% of the websites in the search results were relevant and greater than half (52.4%) of the searches yielded no relevant results in the top 3 results. In comparison with Dothan, Los Angeles websites were more likely to be relevant (OR = 28.1; 95% CI = 10.80, 73.04). Searches for larger cities (Los Angeles, Atlanta, DC) were more likely to yield relevant results compared with small cities (Burlington, Dothan, Las Cruces) (OR = 10.0; 95% CI = 5.6, 17.9).
Regarding domain type, more relevant websites were returned for .org (66.0%) and .gov (65.0%), websites. Fewer than half of the .com websites (40.0%) and .net (29.4%) sites were relevant. Compared with .com domains, .gov (OR = 2.9, 95% CI = 1.4, 5.6) and .org domains (OR = 2.9, 95% CI = 1.7, 4.8) were more likely to retrieve relevant websites. On comparing websites with information about public clinic or hospital services, the following types of websites were significantly less likely to yield relevant information: online directories (OR = .17; 95% CI = .06, 0.47), nonprofit organizations (OR = .244; 95% CI = .085, 0.703), general information resources (OR = .04; 95% CI = 0.01, 0.14), or message boards/blogs (OR = .05; 95% CI = 0.01, 0.30). A statistically significant difference was also found by rank order of the websites (χ2 = 12.12, P < 0.01), with the first search result yielding a relevant website the least often (38.7%) compared with the second search result (51.2%) and the third (60.6%). This finding is likely related to the fact that .org or .gov websites more often yielded relevant information than .com websites, but they less likely appear as the first search result. Indeed, both .org and .gov websites were more commonly ranked second or third search result in Google results (Table 2).
To our knowledge, this is the first study to examine online access to local sexual health services, searching by multiple cities and geographic locations. Results from this analysis support previous research,11,18 documenting the gaps that exist when searching for local sexual health services online. One gap identified in this study is a lack of government-sponsored sites retrieved during searches and the infrequency in which they appear as the first result. This is problematic, as government-sponsored sites often provide more relevant information.18 According to a state directory19, about half of all state health department sites are classified as .gov; therefore, health departments and/or STD programs should strive to achieve a greater online presence so that their sexual health information and services can be more readily found. With many young people engaged in online health information seeking, health departments should create accessible, user-friendly web sites to serve this population.
Moreover, this study revealed that the relevancy of search results largely depends upon the queried health topic, and we speculate that general searches may produce better results. For instance, this research found that more relevant web pages were retrieved when using the general search phrase (STD test) compared with specific phrases (e.g., Chlamydia test and Herpes test). Consequently, search engine users may fare better starting out their queries with general/basic terms and phrases, or without a location attached.
Another important finding from this research pertains to national directories of STD/HIV testing resources. One such directory, www.hivtest.org, hosted by the Centers for Disease Control and Prevention, is an online tool for locating STD/HIV testing centers by ZIP code. Despite using search terms similar to hivtest.org, this testing locator website only appeared twice in our search results. Likewise, the website www.iwantthekit.org, where men and women can order free at-home Chlamydia, Gonorrhea, and trichomoniasis testing kits, never appeared during Chlamydia or STD test searches for Washington, DC, one of the site's service areas. Substantial effort and resources have been devoted to building and maintaining these useful websites, thus it is unfortunate that users may experience difficulties in locating them online. In an effort to promote their resources, website developers/managers should make greater efforts to ensure placement higher in standard Google search results. Moreover, perhaps large organizations, with national reputation and presence, should work harder to make their STD/HIV testing efforts felt on the ground and web.
Findings from this study are subject to several limitations. First, this study was only meant to examine whether STD testing information could be located from a standard Google search. We did not measure the quality of websites; therefore, we cannot conclude that the clinics, phone numbers, or hotlines were valid or that the services they provided actually existed at the time of our queries. Our purpose was to gauge the ease in which people searching online could access local information (accurate or not) relevant to their queried topic. However, time, resources, and institutional review board issues limited us from investigating the accuracy of websites included in our sample. Although website inaccuracies do exist, 1 study found relatively low inaccuracy among 177 websites providing STI/HIV information, and no inaccuracies for HIV testing websites.11 Nonetheless, we believe that website accuracy is always important and recommend that the accuracy of websites be assessed in future research. Additionally, our results may have been distorted because we did not analyze any of the sponsored links usually appearing at the top of Google search result pages. Some research11 has shown that young Internet users click on such sponsored results. However, sponsored links were not included because, in preparing for the study, 2 researchers (in 2 different geographic locations) performed the same search at the same time, retrieving the same result output, but different sponsored links. Because this was a convenience sample, generalizations from findings here to the larger US population or geographic areas are not possible. Finally, this study is only a snapshot of what appears during a standard Google search. If we were to perform the searches today, the same sites may not appear.
This research suggests that it is difficult to find information pertaining to local sexual health services on the Internet. This work reflects previous studies on web search strategies and points to the critical implication that public health clinics, local health departments, and community-based organizations must optimize their website placement so as to reach a greater proportion of the sexually active population who use web search engines. Although an in-depth discussion of search engine optimization is beyond the scope of this article, given our findings, we recommend that STD organizations investigate options for improving their websites' visibility through search engine optimization.
Also, while the larger cities in this study fared better in terms of service information, Burlington is an example that small communities can successfully publish their information in cyberspace. Burlington, our smallest city, achieved only slightly fewer relevant sites than Washington, DC, and Atlanta, GA. To understand how Vermont's health department and nonprofit websites have been so successful as well as how others can replicate their model should be further explored. Finally, this study found that less than one-fifth of websites advertising STD/HIV testing services provided information in a language other than English, providing evidence that sexual health information on the Internet may be inaccessible to groups such as Hispanics, the largest minority group in the United States.20 Considering that Hispanics in the United States suffer disproportionately higher rates of chlamydia, gonorrhea, and syphilis (compared with whites),21 research is sorely needed to understand how culturally and linguistically appropriate STD and sexual health online information can be made more accessible to Hispanics.
In closing, new data from the Pew Internet and American Life Project reveals that 29% of young adults (18–29) are now using cell phones to search for health information online which also may warrant further exploration.22 Likewise a recent news article reported that according to Yahoo!, three out of the five most popular health searches on cell phones pertain to sex: “pregnancy,” “herpes”, and “STD”.23 These search terms do not top the lists on computer-initiated web searches. Researchers speculate that because cell phones offer immediate access to the mobile web, these could be “morning after” searches by individuals worried about an STD infection or pregnancy.23 These findings warrant further investigation into the ease of using cell phones to locate online sexual health information and whether the caliber of retrieved results help expedite care and treatment.
With Internet use ubiquitous among Americans, it is inevitable that health care seekers will rely on search engines to locate readily accessible information online. Accordingly, it is imperative that STD-serving institutions strive to stay as technologically advanced as possible so as to meet the needs of their communities.
1. McMullan M. Patients using the internet to obtain health information: How this affects the patient-health professional relationship. Patient Educ Couns 2006; 63:24–28.
2. Friedman AL, Shepeard H. Exploring the knowledge, attitudes, beliefs, and communication preferences of the general public regarding HPV: Findings from CDC focus group research and implications for practice. Health Educ Behav 2007; 34:471–485.
4. Buhi ER, Daley EM, Oberne A, et al. Quality and accuracy of sexual health information websites visited by young people. J Adolesc Health 2010; 47:206–208.
5. Weinstock H, Berman S, Cates W. Sexually transmitted diseases among American youth: Incidence and prevalence estimates, 2000. Perspect Sex Reprod Health 2004; 36:6–10.
10. Pan B, Hembrooke H, Joachims T, et al. In Google we trust: Users' decisions on rank, position, and relevance. J Comput Mediat Commun 2007; 12:801–823.
11. Buhi ER, Daley EM, Fuhrmann HJ, et al. An observational study of how young people search for online sexual health information. J Am Coll Health 2009; 58:101–111.
14. Centers for Disease Control and Prevention. Sexually Transmitted Disease Surveillance, 2008. Atlanta, GA: US Department of Health and Human Services, 2009.
15. Eysenbach G, Kohler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in depth interviews. BMJ 2002; 324:573–577.
16. Spink A, Wolfram D, Jansen MB, et al. Searching the web: The public and their queries. J Am Soc Info Sci Tech 2001; 52:226–234.
17. Neuendorf KA. The Content Analysis Guidebook. Thousand Oaks, CA: Sage, 2002.
18. Tietz A, Davies S, Moran JS. Guide to sexually transmitted disease resources on the internet. Clin Infect Dis 2004; 38:1304–1310.
19. Centers for Disease Control and Prevention. STD Awareness Month Website, 2010. Available at: www.cdcnpin.org/stdawareness
. Accessed June 11, 2010.
21. Centers for Disease Control and Prevention. Sex Transmitted Diseases in the United States, 2008 National Surveillance Data for Chlamydia, Gonorrhea, and Syphilis. Available at: www.cdc.gov/std/stats08/2008survFactSheet.PDF
. Accessed June 11, 2010.