Share this article on:

Using Twitter to Identify and Respond to Food Poisoning: The Food Safety STL Project

Harris, Jenine K. PhD; Hawkins, Jared B. PhD; Nguyen, Leila MPH; Nsoesie, Elaine O. PhD; Tuli, Gaurav PhD; Mansour, Raed MS; Brownstein, John S. PhD

Journal of Public Health Management & Practice: November/December 2017 - Volume 23 - Issue 6 - p 577–580
doi: 10.1097/PHH.0000000000000516
Research Reports: Research Brief Report

Context: Foodborne illness affects 1 in 4 US residents each year. Few of those sickened seek medical care or report the illness to public health authorities, complicating prevention efforts. Citizens who report illness identify food establishments with more serious and critical violations than found by regular inspections. New media sources, including online restaurant reviews and social media postings, have the potential to improve reporting.

Objective: We implemented a Web-based Dashboard (HealthMap Foodborne Dashboard) to identify and respond to tweets about food poisoning from St Louis City residents.

Design and Setting: This report examines the performance of the Dashboard in its first 7 months after implementation in the City of St Louis Department of Health.

Main Outcome Measures: We examined the number of relevant tweets captured and replied to, the number of foodborne illness reports received as a result of the new process, and the results of restaurant inspections following each report.

Results: In its first 7 months (October 2015-May 2016), the Dashboard captured 193 relevant tweets. Our replies to relevant tweets resulted in more filed reports than several previously existing foodborne illness reporting mechanisms in St Louis during the same time frame. The proportion of restaurants with food safety violations was not statistically different (P = .60) in restaurants inspected after reports from the Dashboard compared with those inspected following reports through other mechanisms.

Conclusion: The Dashboard differs from other citizen engagement mechanisms in its use of current data, allowing direct interaction with constituents on issues when relevant to the constituent to provide time-sensitive education and mobilizing information. In doing so, the Dashboard technology has potential for improving foodborne illness reporting and can be implemented in other areas to improve response to public health issues such as suicidality, spread of Zika virus infection, and hospital quality.

Brown School, Washington University in St Louis, St Louis, Missouri (Dr Harris); Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts (Drs Hawkins, Tuli, and Brownstein); Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (Drs Hawkins and Brownstein); City of St Louis Department of Health, St Louis, Missouri (Ms Nguyen); Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington (Dr Nsoesie); and Chicago Department of Public Health, Chicago, Illinois (Mr Mansour).

Correspondence: Jenine K. Harris, PhD, Brown School, Washington University in St Louis, One Brookings Dr, St Louis, MO 63130 (harrisj@wustl.edu).

Jenine Harris received pilot funding from the Center for Dissemination and Implementation at Washington University in St Louis to support a small percentage of her time and a research assistant to help implement and evaluate the Dashboard. Elaine Nsoesie and John Brownstein received a grant from National Library of Medicine of the National Institutes of Health (award no. R01LM011965) to support the Dashboard development work including supporting the time of Jared Hawkins and Gaurav Tuli. Leila Nguyen and Raed Mansour have no funding sources associated with this work. The funders did not have any role in study design, collection, analysis, or interpretation of data, writing the report, or the decision to submit for publication.

The authors acknowledge and thank Roger Wong, who worked with the team to implement the Dashboard and reply to tweets. The authors are thankful for pilot funding from the Center for Dissemination and Implementation at Washington University in St Louis and a grant from National Library of Medicine of the National Institutes of Health (award no. R01LM011965). The authors also thank Carl Filler for helping them start the process of getting the Dashboard implemented, Candace Da Silva for assisting with food inspection data, and their many colleagues at the City of St Louis Department of Health who were instrumental and enthusiastic in integrating this new technology into existing systems.

The authors declare no conflicts of interest.

Supplemental digital content is available for this article. Direct URL citation appears in the printed text and is provided in the HTML and PDF versions of this article on the journal's Web site (http://www.JPHMP.com).

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Foodborne illness affects 1 in 4 Americans and costs approximately $2 billion to $4 billion annually in the United States.1 Although the Centers for Disease Control and Prevention classifies foodborne illness as a winnable battle,2 or a priority “with large-scale impact on health and known effective strategies to address [it],” as few as 2.9% of those who become sick seek medical care and most do not report their illness.3,4 As a result, existing systems capture a small fraction of the disease burden, making prevention and evaluation of policies and programs challenging.3,5

Local health departments (LHDs) nationwide license and inspect restaurants to ensure food safety and respond to reports of suspected foodborne illness. Restaurants with lower inspection scores have higher foodborne illness outbreak rates,6 and inspections prompted by consumer reporting tend to identify more serious and critical violations than regular inspections.7 In 2012-2013, the New York City Department of Health and Mental Hygiene used Yelp reviews to identify restaurants where patrons became ill.7 In 2013-2014, the Chicago Department of Public Health (CDPH) used Twitter to identify and respond to food poisoning tweets.8 These programs were successful in identifying and addressing food safety violations,7,8 providing evidence of the potential of new media to improve foodborne illness reporting.

In St Louis, reporting suspected food poisoning to the Department of Health (STL-DOH) can happen in 7 ways: a phone call, tweet, fax, or e-mail to the Citizen Service Bureau, direct contact with inspectors by another health department or source, or completing a form on the STL-DOH Web site. These reporting mechanisms resulted in 46 consumer reports in 2014 from a city of 318 000 residents.

In 2015, the STL-DOH partnered with Washington University in St Louis to implement the HealthMap Foodborne Dashboard (Dashboard) developed at Boston Children's Hospital. The Dashboard monitors Twitter for food poisoning tweets globally but can focus on a specific geographic area. This evaluation examines the performance of the Dashboard after implementation in St Louis.

Back to Top | Article Outline

Methods

We set the Dashboard to capture tweets including “food poisoning,” “foodpoisoning,” or both, and to classify them as relevant, unclear, or not relevant using a supervised machine learning classifier. The 2 terms were selected on the basis of informal tests during development that found these terms together captured most relevant tweets, with the fewest irrelevant tweets. Starting October 2015, the Dashboard displayed machine-classified tweets within a 50-mile radius around St Louis and a human reviewed each tweet to verify or change the machine classification. Tweets were reviewed and classified daily; the Table shows example tweets.

After machine and human classification, we used a reply feature in the Dashboard to construct a reply to each tweet human classified as relevant or unclear; reviewing, classifying, and replying to a tweet takes less than 1 minute. Following work by CDPH,9 reply tweets included an empathy statement expressing concern for the person (eg, Sorry to hear you're ill), an authority statement identifying the official source of the tweet (eg, The City can help you report it), and a call-to-action (eg, Click here) with a link to a form for reporting illness to STL-DOH. Here's an example of a full reply: Sorry to hear you're ill. The City can help you report it. Click here: [URL]. The @FoodSafetySTL Twitter account used to send reply tweets is an official account linked to the STL-DOH Web site in the Twitter profile.10 Information from reports submitted through the form was treated the same as information from other reporting mechanisms. This study was approved by the institutional review board at Washington University in St Louis (ID#201601097).

Back to Top | Article Outline

Results

In its first 7 months of operation, the Dashboard identified 442 tweets, or approximately 2 tweets per day, within the radius and including 1 or more food poisoning terms. Of these, the machine classified 214 tweets as relevant, 24 as unclear, and 204 as not relevant (Table). Human coding agreed for 66.4% of relevant tweets, 8.3% of unclear tweets, and 77.5% of not relevant tweets. A human classified 217 tweets as relevant or unclear; 24 of these were not in St Louis or were from a user who had tweeted and been replied to recently, so 193 tweets received a reply. Thirteen replies, or 6.7% of tweets replied to, resulted in a report submission. Five of the 13 reports were in the STL-DOH jurisdiction, whereas 8 were in neighboring jurisdictions and were forwarded to those jurisdictions. During this same time frame, the STL-DOH received 31 other foodborne illness reports submitted by fax or direct tweet (none), the existing Web form (n = 3), directly from the county health department (n = 3), directly from other sources (n = 3), through e-mail (n = 6), and by phone (n = 16) (see Figure 1 Supplemental Digital Content, available at http://links.lww.com/JPHMP/A280).

The STL-DOH inspects all restaurants reported by patrons as a possible source of illness. One restaurant was identified in 2 reports (1 phone and 1 e-mail report) on the same day and was only inspected once, another report had no evidence of illness, and 1 inspection was not entered into the database properly, so the score was unclear. The 33 total inspections completed and entered during the study period included 5 from Dashboard forms and 28 from other sources. Of the 5 via the Dashboard, 2 (40%) had no violations, 2 (40%) had noncritical violations, and 1 (20%) had noncritical and critical violations. Of the 28 other inspections, 7 (25%) had no violations, 10 (36%) had noncritical violations, and 11 (39%) had noncritical and critical violations. A Fisher exact test found no significant difference in the proportion of restaurants with and without violations (P = .60) reported via the Dashboard compared with other reporting methods.

Back to Top | Article Outline

Discussion

In our pilot study, we found the Dashboard to have the potential for improving foodborne illness reporting through increased citizen engagement. In the first 7 months of Dashboard use, the only mechanisms resulting in more reports than the Dashboard were phone and e-mail. The large number of phone reports is consistent with research identifying phone numbers and Web forms as recommended ways to increase foodborne illness reporting by the public.4 Inspection results were not significantly different for reports originating with the Dashboard compared with other reports.

We identified a few limitations during this pilot phase to address as we work with existing and new partners on Dashboard implementation in health departments. First, we observed disagreement between human and machine classification. This likely happened because the Dashboard was designed to label and display all tweets fitting a broad range of search criteria rather than setting a stricter threshold and missing relevant tweets, and the process of developing a reliable machine learning classifier is iterative implying that as more data become available, the accuracy of the machine learning labeling should improve. We are examining the features of the misclassified tweets and will use the information to retrain the classifier with these features in order to improve machine-human agreement. Second, St Louis City borders St Louis County, which is home to many restaurants. We forwarded complaints about county restaurants to the county health department but were not able to track their progress. In future efforts, we will encourage Dashboard partnerships in geographic areas served by multiple LHDs. In addition, while government agencies in urban areas often implement innovative public health programs through new technology,11 the project was the first of its kind in St Louis City and recipients of a @FoodSafetySTL tweet may have been skeptical, given the lack of a local track record. As adoption of new technology for citizen engagement in public health increases, we expect @FoodSafetySTL trust and engagement to also increase.12,13 The Dashboard is still in use by the STL-DOH; you can find @FoodSafetySTL on Twitter and more information on the STL-DOH Web site.

The Dashboard differs from other citizen engagement mechanisms in its use of current data, allowing interaction with constituents on issues when the issue is relevant to the constituent to provide education and mobilizing information. Prior studies have demonstrated that the interactive and real-time nature of social media benefits government and the public during emergencies and for everyday concerns.12 Government use of social media also improves perceived transparency and increases trust in government,13 resulting in increased public interaction with government. Given extremely low reporting rates for foodborne illness3,4 and limited public knowledge of reporting processes,4 increasing trust and interaction between government and the public through social media are promising strategies for food safety. About 23% of the US population uses Twitter, including more urban residents, men, college graduates, and higher-income earners, but fewer whites or adults aged 65+ years.14 Extending the Dashboard to other social media could also improve reporting among non-Twitter users.

The Dashboard technology also has potential to aid in educating the public by retweeting food safety tweets captured by the Dashboard and alerting citizens to the process of reporting foodborne illness to their health department. In addition, there are uses for the Dashboard beyond food safety. Dashboard technology has been adapted to examine tweets about hospital quality,15 and a similar approach has been used to identify tweets associated with suicidality.16 The Dashboard could also be used to alert health departments of local tweets about concerns such as standing water that encourages spread of Zika virus infection in a community.

Back to Top | Article Outline

Implications for Policy & Practice

Foodborne illness is vastly underreported by the U.S. public but new media like Twitter and Yelp have the potential to improve reporting. Our pilot project evaluated the use of a new Twitter tool by the City of St. Louis Department of Health to identify and respond to food poisoning tweets in the local area. We found:

• Constituents submitted reports after being contacted via Twitter at a higher rate than most other existing foodborne illness reporting mechanisms.

• Inspection results following reports from tweeters were consistent with inspection results from other types of reporting.

• There was room for improvement in the system used to identify relevant tweets.

• Geographically proximate health departments may wish to partner on Twitter use for food safety.

Back to Top | Article Outline

References

1. McCabe-Sellers BJ, Beattie SE. Food safety: emerging trends in foodborne illness surveillance and prevention. J Am Diet Assoc. 2004;104:1708–1717.
2. Centers for Disease Control and Prevention. Winnable Battles. http://http://www.cdc.gov/winnablebattles Updated 2015. Accessed April 14, 2015.
3. Mead PS, Slutsker L, Dietz V, et al Food-related illness and death in the United States. Emerg Infect Dis. 1999;5:607–625.
4. Arendt S, Rajagopal L, Strohbehn C, Stokes N, Meyer J, Mandernach S. Reporting of foodborne illness by US consumers and healthcare professionals. Int J Environ Res Public Health. 2013;10:3684–3714.
5. Li J, Smith K, Kaehler D, Everstine K, Rounds J, Hedberg C. Evaluation of a statewide foodborne illness complaint surveillance system in Minnesota, 2000 through 2006. J Food Protect. 2010;73:2059–2064.
6. Irwin K, Ballard J, Grendon J, Kobayashi J. Results of routine restaurant inspections can predict outbreaks of foodborne illness: the Seattle-King County experience. Am J Public Health. 1989;79:586–590.
7. Harrison C, Jorder M, Stern H, et al Using online reviews by restaurant patrons to identify unreported cases of foodborne illness—New York City, 2012-2013. MMWR Morb Mortal Wkly Rep. 2014;63:441–445.
8. Harris JK, Mansour R, Choucair B, Olson J, Nissen C, Bhatt J. Health department use of social media to identify foodborne illness—Chicago, Illinois, 2013-2014. MMWR Morb Mortal Wkly Rep. 2014;63:681–685.
9. Marziano S. Cutgroup #9—Foodborne Chicago. http://http://www.smartchicagocollaborative.org/cutgroup-9-foodborne-chicago Updated 2014. Accessed May 24, 2016.
10. City of St Louis Department of Health. Food safety STL. https://http://www.stlouis-mo.gov/government/departments/health/environmental-health/food-control/food-safety-stl.cfm. Updated 2016. Accessed May 28, 2016.
11. National Association of County & City Health Officials. National Profile of Local Health Departments. http://nacchoprofilestudy.org. Updated 2014. Accessed July 10, 2016.
12. Veil SR, Buehner T, Palenchar MJ. A work‐in‐process literature review: incorporating social media in risk and crisis communication. J Contingencies Crisis Manage. 2011;19:110–122.
13. Song C, Lee J. Citizens' use of social media in government, perceived transparency, and trust in government. Public Perform Manage Rev. 2016;39:430–453.
14. Duggan M, Ellison NB, Lampe C, et al A Social Media Update 2014. Washington, DC: Pew Research Center; 2015:1–17.
15. Hawkins JB, Brownstein JS, Tuli G, et al Measuring patient-perceived quality of care in US hospitals using Twitter. BMJ Qual Saf. 2016;25:404–413.
16. Ung J. ARKHumanity app saves lives one tweet at a time. The State Press. September 17, 2015. http://http://www.statepress.com/article/2015/09/ark-humanity-app-save-lives. Accessed September 6, 2017.

food safety; local health department; Twitter

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.