Goto, Yasushi MD*; Sekine, Ikuo MD, PhD*; Sekiguchi, Hiroshi MD†; Yamada, Kazuhiko MD*; Nokihara, Hiroshi MD*; Yamamoto, Noboru MD*; Kunitoh, Hideo MD*; Ohe, Yuichiro MD*; Tamura, Tomohide MD*
The Internet has given rise to an information revolution of unprecedented magnitude. Whereas the Internet has great potential in marshaling the large volume of health information resources available, it is becoming increasingly difficult to discern which of the resources are reliable and accurate or appropriate for the users.1–6 This issue has become a cause for great concern, especially in the field of oncology, and many studies have evaluated the pros and cons of obtaining information from the Internet.2–6 Meanwhile, the medical community is being increasingly faced with patients asking us about the medical information available on the Internet. We can no longer neglect the public importance of the information available and have to use it effectively for patients to better understand their disease.
Although one of the main characteristics of the Internet is its worldwide accessibility, differences in language use around the world serve as a bottleneck for collecting information from the Internet. The estimated number of people using the Internet is about the same in the United States and Japan (70 and 67%,7,8 respectively), and 80% of patients obtain health information via the Internet in the United States.9 Until now, most studies that have evaluated the quality of the health care information available over the Internet are from the English-speaking community, and very few studies have been conducted in relation to information available in Japanese.10,11 Furthermore, only a limited number of studies evaluating the differences in the quality of information available between two languages have been published,12 and no such study comparing such information in the English and Japanese languages has been published.
Our goal was to imitate the search for medical information by the general population in Japan and United States and to evaluate the differences in the process between the two countries. We also investigated the administrators of the Web sites and attempted to identify any correlation existing between the Web site administrators and the quality of information available on the Internet. We focused on information available on lung cancer, which is the leading cause of cancer-related death in both the Unites States and Japan.13,14 Because search engines are the leading tools to obtain any kind of information, whether general or medical, on the Internet,15 we used Google and Yahoo, which are the two most commonly used search engines for Web search in both the United States and Japan.
Web Site Search
We conducted a prospective, observational Web review by performing keyword searches using Google in both Japanese and English, and Yahoo in Japanese. Japanese searches were conducted by author YG in Japan (Tokyo) on May 29, 2007, and the English search was conducted by author HS in the United States (New York) on May 25, 2007. We used “Hai-gan (both letters in Chinese characters),” “Hai (Chinese character)-gan (hiragana),” and “Hai (Chinese character)-gan (katakana),” for the Japanese search, and “lung cancer” and “lung carcinoma” for the English search. The search word that resulted in the largest number of search results was chosen for the subsequent study.
The first 50 Web sites displayed by Google and Yahoo in Japanese, and Google in English, excluding the advertisement area, were used for further evaluation. Web sites that were inaccessible, not designed to provide health information (i.e., news and advertisement of books), or displayed for the second (or more) time were excluded from the subsequent evaluation. Samples from the Yahoo in English were supplemented to compare the search utility on January 21, 2009.
Author YG evaluated the Web sites within a week of the original search. We evaluated the Web sites based on criteria known as the “JAMA” benchmark16: display of authorship (authors and contributors, their affiliations, and relevant credentials), attribution (references and sources for all content and all relevant copyright information), disclosure (Web site ownership, sponsorship, advertising, commercial funding arrangements or support, or potential conflicts of interest), and currency (dates on which the contents were posted and updated). We considered each criterion as fulfilled when it was fully displayed. For further evaluation, we focused on the description about the treatment of advanced non-small lung cancer. To our knowledge, there is no established tool-based instrument to evaluate the information available on cancer treatment. Therefore, we classified the information into three categories: acceptable (description of systematic reviews, such as guidelines from authorized facilities,17–20 links to systematic reviews, or abstracts of systematic reviews), unacceptable (recommendation of alternative medicine or a generally unapproved treatment), and inevaluable (lack of adequate description). The administrators of the Web sites were classified into five categories: nonprofit organization (NPO) or public institution, medical institution, commercial (for specific treatments), personal (pages made by patients or their families), and others.
Descriptive statistics were used to determine the numbers and percentages related to the characteristics of the Web sites. To compare the differences between two countries in view of user experience and search utility, Web sites displayed in Google-U was compared with that of Yahoo-J and Google-J, respectively. The χ2 test or Fisher's exact test was used as appropriate.
Differences by Notation
In Google Japan, search using the word “Hai-gan (both letters in Chinese characters)” resulted in a display of approximately 7.7 million Web sites, and in Google United States, search using the phrase “lung cancer” threw up approximately 52 million Web sites. These notations were, therefore, used for the subsequent evaluation. After excluding Web sites that were inaccessible, were not designed to provide health information, or ranked for the second (or more) time in each search, 44, 27, 39, and 35 Web sites displayed by Yahoo Japan (Yahoo-J), Google Japan (Google-J), Yahoo United States (Yahoo-U), and Google United States (Google-U), respectively, were evaluated for further study.
Web Site Characteristics
Figure 1 summarizes the quality of the Web sites that satisfied the criteria of the JAMA benchmark. Authorship was displayed in more than 70% of the Web sites displayed by the three searches: 31 in Google-U (88.6%), 34 in Yahoo-J (70.3%, p = 0.243), and 19 in Google-J (88.6%, p = 0.106). Attribution of the content was found in 20 (57.1%) of the Web sites in Google-U, and 7 (15.9%, p < 0.001) and 6 (22.2%, p = 0.009) of the Web sites in Yahoo-J and Google-J, respectively. Twenty-eight (63.6%, p = 0.001) Web sites in Yahoo-J, 11 (40.7%, p < 0.001) in Google-J, and 33 (94.2%) in Google-U made the disclosure. Display of currency was found in 29 (82.9%) sites in Google-U, but in less than 50% of the Web sites in the Japanese searches; 11 (25.0%, p < 0.001) in Yahoo-J and 11 (40.7%, p = 0.001) in Google-J.
Quality of Description of the Treatment
Evaluation of the treatment description for advanced non-small cell lung cancer is summarized in Figure 2. The description was acceptable in 28 (80.0%) of the Web sites generated by Google-U, as these sites described chemotherapy as the standard treatment for advanced lung cancer. Only one site recommended alternative medicine. In Web sites ranked by Yahoo-J and Google-J, standard therapy was only described in 20 (45.5%, p < 0.001) and 10 (37.0%, p < 0.001) sites, respectively, whereas 7 (15.9%, p = 0.070) and 7 (25.9%, p = 0.017) sites, respectively, recommended alternative medicine. Table 1 summarizes the quality of the Web sites displayed in Yahoo and Google by both countries. Proportions of reliable sites were comparable in countries but were not in search engines.
Administrators of the Web sites
The administrators of the Web sites are shown in Figure 3. In Google-U, the administrators of 16 (45.7%) Web sites were NPO or public institution, whereas only 7 (15.9%, p = 0.006) and 2 (7.4%, p = 0.001), respectively, in Yahoo-J and Google-J were managed by them. Commercial site for specific treatments was not displayed in Google-U but was displayed in 8 (18.2%, p = 0.007) and 6 (22.2%, p = 0.005) Web sites in Yahoo-J and Google-J, respectively. Web sites administered personally by the patients themselves or their families were also not found among the Web site displayed in Google-U, whereas 4 (9.1%, p = 0.125) sites in Yahoo-J and 7 (25.9%, p = 0.002) sites in Google-J were personally managed.
Administrators and Quality of the Contents of the Web Sites
Table 2 shows the correlation between the Web site administrator and the quality of the contents of the sites. Ten sites generated by both Google-J and Yahoo-J were integrated. There was no site from NPO or public institution category, either Japanese or English, which provided misleading information. Most of the unacceptable sites were managed by commercial or personal sites, neither of which was found in the English-language sites.
By comparing the differences of quality of cancer information on the Internet between the different languages, we, for the first time, evaluated the correlation between the Web site administrator and the quality of the medical information in the Web sites. Furthermore, it is one of the few studies to evaluate the information on lung cancer available on the Internet.15 We also showed that the Web sites displayed in the United States provide information of much higher quality than those displayed by Japanese Web sites, with regard to lung cancer treatment, and this may be related to the quality of the administrators of the displayed Web sites.
It is generally a difficult task to make people access reliable Web sites that would provide the precise information that they are looking for. Regulating access to only trustworthy Web sites that provide useful information is extremely difficult, because a global rule is a necessary step toward controlling the content of the worldwide Web sites. There are also no confirmed tools for weighting the information on the Internet in any field, including medicine. In this chaotic scenario, search engines such as Google and Yahoo have come up with a solution by developing an algorithm to rank the sites. Nowadays, their value is well established in the Internet, and people are generally using this tool for searching medical and other information. Even though there is a concern that the order in which the sites are placed by these tools is not entirely appropriate for the field of medicine,3,21,22 the high frequency at which these are used has made it meaningless to say that they pose a problem in one-particular field. Therefore, what we must consider now is how to provide reliable information using these tools.
Why is misleading and nonreliable information provided on the Internet? One key characteristic of the Internet is the interaction between the provider and the consumer (in the medical field, patient). Web sites that are not accessed frequently will be ranked lower in the search engine system. Therefore, when discussing the results of Web sites ranked by the search engine, we should consider it from both the standpoint of the provider and the consumer. People access the Internet by requesting the information they want. Many cancer patients suffer from an incurable disease and look for a ray of hope in the Internet. This situation is most advantageous to the information senders. They can promote their treatment as the treatment that would bring about the miraculous cure that the patients are seeking. In this study, most of the sources recommending alternative or unapproved drugs were from commercial and personal sites. Information on medical subjects should be correct and be of assistance to the users to help them better understand their disease. People should be protected from disruptive information. Creating confusion in the minds of people by providing misleading information for profit to the administrator is a vexing situation.
One of the interesting findings in this study was that the correlation between the quality of the Web site administrator and the quality of the contents of the site was seen not only for sites providing misleading information but also for those providing reliable information. At present, there are two major administrators providing reliable information, namely, medical institutions and specialized organizations for information administered by patient advocate NPO or public institution. However, the type of information provided differed between the two types of administrators. In general, each medical institution provides reliable messages but not review articles, whereas the patient advocate group NPO and public institution provide a path to the review articles. This is not surprising because the aims of providing information are different between the two types of administrators. For each medical institution, the goal is to display the treatment that they are interested in, and describing the entire medical consensus is outside their reach. Therefore, sites specialized in providing information are the ones that can be most expected to provide general information. Differences in the number of reliable sites between the languages in this study may be because of the difference in the number of such organizations between the countries. The number of public institution sites may depend on the countries in which each language is spoken in, and the growth in the number of patient advocate NPO may depend on the social system or the differences in culture. However, it is noteworthy that patient advocate NPO can play a major role in providing reliable health information.
There were several limitations in this study. One is that we evaluated sites only from Yahoo Japan and Google Japan, and Google United States. We chose Google United States as the reference, because most previous studies on the Internet have been conducted in the United States, and Google is the most popular search engine in the United States.23 In Japan, Yahoo ranks first as the most frequently used search engine, followed next by Google,24 which is the reason we selected these two as the representative search engines for our search of Web sites in Japanese. Although this approach may limit evaluation of the overall Internet situation in the two countries, we believe that this was the closest way to reproduce the way people browse the Internet. Another concern is the number of sites generated by these tools. The total number of Web sites displayed by our search using the keywords differs between the two languages and maybe attributable to the differences in the quality of the administrators. Google-U generated approximately seven times as many Web sites as Google-J. This discrepancy could be because of the difference in the number of people using the two languages. However, we only evaluated the top 50 sites, which is far short of the total number of sites displayed but may already be too much for anyone seeking any type of information. Because the ranking system has prevailed, the quality of the highest ranked Web sites and not the total number of sites displayed is important to the user. Lastly, another important problem is whether people in the United States and Japan desire the same answers from the Internet. In general, search engines attempt to rank the Web sites sought by the users. If these differed between countries, the ranking would also reflect these differences. Differences in the social backgrounds of the populations in the two countries were confounding factors in this study. However, no studies evaluating the topic from this perspective have been conducted. These are topics of interest that need further investigation.
In this era of abundance of information, it is absolutely essential for people to make their choices based on the quality. As medical professionals, we have the responsibility of providing appropriate information to people who are unaware and anxious about their future. In the new era of the Internet technology, facilitating easy access to reliable information, and providing reliable information is important. This study may facilitate an understanding of the actual status of dispersal of information and pave the way for discussing methods to achieve better accessibility to high-quality health information.
1. Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. BMJ 2002;324:598–602.
2. Bichakjian CK, Schwartz JL, Wang TS, Hall JM, Johnson TM, Biermann JS. Melanoma information on the internet: often incomplete—a public health opportunity? J Clin Oncol 2002;20:134–141.
3. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. BMJ 2002;324:577–581.
4. Helft PR, Hlubocky F, Daugherty CK. American oncologists' views of internet use by cancer patients: a mail survey of American Society of Clinical Oncology members. J Clin Oncol 2003;21:942–947.
5. Trumbo CW. Cancer information on the world wide Web: gross characteristics. J Natl Cancer Inst 2004;96:332–333.
6. Helft PR, Eckles RE, Johnson-Calley CS, Daugherty CK. Use of the internet to obtain cancer information among cancer patients at an urban county hospital. J Clin Oncol 2005;23:4954–4962.
10. Nemoto K, Tachikawa H, Sodeyama N, et al. Quality of Internet information referring to mental health and mental disorders in Japan. Psychiatry Clin Neurosci 2007;61:243–248.
11. Tatsumi H, Mitani H, Haruki Y, Ogushi Y. Internet medical usage in Japan: current situation and issues. J Med Inter Res 2001;3:E12.
12. Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285:2612–2621.
13. Minino AM, Heron MP, Murphy SL, Kochanek KD. Deaths: final data for 2004. Natl Vital Stat Rep 2007;55:1–119.
15. Linssen C, Schook RM, The AM, Lammers E, Festen J, Postmus PE. A web site on lung cancer: who are the users and what are they looking for? J Thorac Oncol 2007;2:813–818.
16. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor—Let the reader and viewer beware. JAMA 1997;277:1244–1245.
17. Pfister DG, Johnson DH, Azzoli CG, et al. American Society of Clinical Oncology treatment of unresectable non-small-cell lung cancer guideline: update 2003. J Clin Oncol 2004;22:330–353.
18. Socinski MA, Morris DE, Masters GA, Lilenbaum R. Chemotherapeutic management of stage IV non-small cell lung cancer. Chest 2003;123:226S–243S.
21. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000;321:1511–1515.
22. Borges AAH, Cervi PM, Arcaya MLTAd, Guardado G, Rabaza AR, Sosa AJ. Rate of compliance with the HON code of conduct versus number of inbound links as quality markers of pediatric web sites. In MEDNET2001;2001, Udine, Italy: Technology and Healthcare, IOS Press Amsterdam, 2001.