Foodborne illness affects 1 in 4 Americans and costs approximately $2 billion to $4 billion annually in the United States.1 Although the Centers for Disease Control and Prevention classifies foodborne illness as a winnable battle,2 or a priority “with large-scale impact on health and known effective strategies to address [it],” as few as 2.9% of those who become sick seek medical care and most do not report their illness.3,4 As a result, existing systems capture a small fraction of the disease burden, making prevention and evaluation of policies and programs challenging.3,5
Local health departments (LHDs) nationwide license and inspect restaurants to ensure food safety and respond to reports of suspected foodborne illness. Restaurants with lower inspection scores have higher foodborne illness outbreak rates,6 and inspections prompted by consumer reporting tend to identify more serious and critical violations than regular inspections.7 In 2012-2013, the New York City Department of Health and Mental Hygiene used Yelp reviews to identify restaurants where patrons became ill.7 In 2013-2014, the Chicago Department of Public Health (CDPH) used Twitter to identify and respond to food poisoning tweets.8 These programs were successful in identifying and addressing food safety violations,7,8 providing evidence of the potential of new media to improve foodborne illness reporting.
In St Louis, reporting suspected food poisoning to the Department of Health (STL-DOH) can happen in 7 ways: a phone call, tweet, fax, or e-mail to the Citizen Service Bureau, direct contact with inspectors by another health department or source, or completing a form on the STL-DOH Web site. These reporting mechanisms resulted in 46 consumer reports in 2014 from a city of 318 000 residents.
In 2015, the STL-DOH partnered with Washington University in St Louis to implement the HealthMap Foodborne Dashboard (Dashboard) developed at Boston Children's Hospital. The Dashboard monitors Twitter for food poisoning tweets globally but can focus on a specific geographic area. This evaluation examines the performance of the Dashboard after implementation in St Louis.
We set the Dashboard to capture tweets including “food poisoning,” “foodpoisoning,” or both, and to classify them as relevant, unclear, or not relevant using a supervised machine learning classifier. The 2 terms were selected on the basis of informal tests during development that found these terms together captured most relevant tweets, with the fewest irrelevant tweets. Starting October 2015, the Dashboard displayed machine-classified tweets within a 50-mile radius around St Louis and a human reviewed each tweet to verify or change the machine classification. Tweets were reviewed and classified daily; the Table shows example tweets.
Example of Foodborne-Related Tweets in Each Class and Number of Machine Classified, Human Classified, and Consistently Classified by Both Machine and Human
||Tweet about a possible current case of foodborne illness
I'm straight hella sick with food poisoning
i think i got food poisoning.
Well, things didn't go my way this weekend in St. Louis. I got some bad food poisoning and couldn't... https://t.co/dmYeuyfG9R
||Unable to distinguish whether tweet is relevant or not relevant
Woohoo possible food poisoning :(
BREAKING: Food Poisoning SUCKS
Chipotle will close restaurants briefly on Feb. 8 to address food safety issues: Successive food poisoning... https://t.co/ZbTgTh77Zm
||Not about a possible current case of foodborne illness
Hope I don't get food poisoning from IKEA.
Food poisoning makes you terrified to eat any food ever again
48 Million People Get Ill from Food Poisoning Says Chef Remi http://t.co/Sj7hepmmwF #pr
After machine and human classification, we used a reply feature in the Dashboard to construct a reply to each tweet human classified as relevant or unclear; reviewing, classifying, and replying to a tweet takes less than 1 minute. Following work by CDPH,9 reply tweets included an empathy statement expressing concern for the person (eg, Sorry to hear you're ill), an authority statement identifying the official source of the tweet (eg, The City can help you report it), and a call-to-action (eg, Click here) with a link to a form for reporting illness to STL-DOH. Here's an example of a full reply: Sorry to hear you're ill. The City can help you report it. Click here: [URL]. The @FoodSafetySTL Twitter account used to send reply tweets is an official account linked to the STL-DOH Web site in the Twitter profile.10 Information from reports submitted through the form was treated the same as information from other reporting mechanisms. This study was approved by the institutional review board at Washington University in St Louis (ID#201601097).
In its first 7 months of operation, the Dashboard identified 442 tweets, or approximately 2 tweets per day, within the radius and including 1 or more food poisoning terms. Of these, the machine classified 214 tweets as relevant, 24 as unclear, and 204 as not relevant (Table). Human coding agreed for 66.4% of relevant tweets, 8.3% of unclear tweets, and 77.5% of not relevant tweets. A human classified 217 tweets as relevant or unclear; 24 of these were not in St Louis or were from a user who had tweeted and been replied to recently, so 193 tweets received a reply. Thirteen replies, or 6.7% of tweets replied to, resulted in a report submission. Five of the 13 reports were in the STL-DOH jurisdiction, whereas 8 were in neighboring jurisdictions and were forwarded to those jurisdictions. During this same time frame, the STL-DOH received 31 other foodborne illness reports submitted by fax or direct tweet (none), the existing Web form (n = 3), directly from the county health department (n = 3), directly from other sources (n = 3), through e-mail (n = 6), and by phone (n = 16) (see Figure 1 Supplemental Digital Content, available at https://links.lww.com/JPHMP/A280).
The STL-DOH inspects all restaurants reported by patrons as a possible source of illness. One restaurant was identified in 2 reports (1 phone and 1 e-mail report) on the same day and was only inspected once, another report had no evidence of illness, and 1 inspection was not entered into the database properly, so the score was unclear. The 33 total inspections completed and entered during the study period included 5 from Dashboard forms and 28 from other sources. Of the 5 via the Dashboard, 2 (40%) had no violations, 2 (40%) had noncritical violations, and 1 (20%) had noncritical and critical violations. Of the 28 other inspections, 7 (25%) had no violations, 10 (36%) had noncritical violations, and 11 (39%) had noncritical and critical violations. A Fisher exact test found no significant difference in the proportion of restaurants with and without violations (P = .60) reported via the Dashboard compared with other reporting methods.
In our pilot study, we found the Dashboard to have the potential for improving foodborne illness reporting through increased citizen engagement. In the first 7 months of Dashboard use, the only mechanisms resulting in more reports than the Dashboard were phone and e-mail. The large number of phone reports is consistent with research identifying phone numbers and Web forms as recommended ways to increase foodborne illness reporting by the public.4 Inspection results were not significantly different for reports originating with the Dashboard compared with other reports.
We identified a few limitations during this pilot phase to address as we work with existing and new partners on Dashboard implementation in health departments. First, we observed disagreement between human and machine classification. This likely happened because the Dashboard was designed to label and display all tweets fitting a broad range of search criteria rather than setting a stricter threshold and missing relevant tweets, and the process of developing a reliable machine learning classifier is iterative implying that as more data become available, the accuracy of the machine learning labeling should improve. We are examining the features of the misclassified tweets and will use the information to retrain the classifier with these features in order to improve machine-human agreement. Second, St Louis City borders St Louis County, which is home to many restaurants. We forwarded complaints about county restaurants to the county health department but were not able to track their progress. In future efforts, we will encourage Dashboard partnerships in geographic areas served by multiple LHDs. In addition, while government agencies in urban areas often implement innovative public health programs through new technology,11 the project was the first of its kind in St Louis City and recipients of a @FoodSafetySTL tweet may have been skeptical, given the lack of a local track record. As adoption of new technology for citizen engagement in public health increases, we expect @FoodSafetySTL trust and engagement to also increase.12,13 The Dashboard is still in use by the STL-DOH; you can find @FoodSafetySTL on Twitter and more information on the STL-DOH Web site.
The Dashboard differs from other citizen engagement mechanisms in its use of current data, allowing interaction with constituents on issues when the issue is relevant to the constituent to provide education and mobilizing information. Prior studies have demonstrated that the interactive and real-time nature of social media benefits government and the public during emergencies and for everyday concerns.12 Government use of social media also improves perceived transparency and increases trust in government,13 resulting in increased public interaction with government. Given extremely low reporting rates for foodborne illness3,4 and limited public knowledge of reporting processes,4 increasing trust and interaction between government and the public through social media are promising strategies for food safety. About 23% of the US population uses Twitter, including more urban residents, men, college graduates, and higher-income earners, but fewer whites or adults aged 65+ years.14 Extending the Dashboard to other social media could also improve reporting among non-Twitter users.
The Dashboard technology also has potential to aid in educating the public by retweeting food safety tweets captured by the Dashboard and alerting citizens to the process of reporting foodborne illness to their health department. In addition, there are uses for the Dashboard beyond food safety. Dashboard technology has been adapted to examine tweets about hospital quality,15 and a similar approach has been used to identify tweets associated with suicidality.16 The Dashboard could also be used to alert health departments of local tweets about concerns such as standing water that encourages spread of Zika virus infection in a community.
Implications for Policy & Practice
Foodborne illness is vastly underreported by the U.S. public but new media like Twitter and Yelp have the potential to improve reporting. Our pilot project evaluated the use of a new Twitter tool by the City of St. Louis Department of Health to identify and respond to food poisoning tweets in the local area. We found:
- Constituents submitted reports after being contacted via Twitter at a higher rate than most other existing foodborne illness reporting mechanisms.
- Inspection results following reports from tweeters were consistent with inspection results from other types of reporting.
- There was room for improvement in the system used to identify relevant tweets.
- Geographically proximate health departments may wish to partner on Twitter use for food safety.
1. McCabe-Sellers BJ, Beattie SE. Food safety: emerging trends in foodborne illness surveillance and prevention. J Am Diet Assoc. 2004;104:1708–1717.
2. Centers for Disease Control and Prevention. Winnable Battles. http://www.cdc.gov/winnablebattles
Updated 2015. Accessed April 14, 2015.
3. Mead PS, Slutsker L, Dietz V, et al. Food-related illness and death in the United States. Emerg Infect Dis. 1999;5:607–625.
4. Arendt S, Rajagopal L, Strohbehn C, Stokes N, Meyer J, Mandernach S. Reporting of foodborne illness by US consumers and healthcare professionals. Int J Environ Res Public Health. 2013;10:3684–3714.
5. Li J, Smith K, Kaehler D, Everstine K, Rounds J, Hedberg C. Evaluation of a statewide foodborne illness complaint surveillance system in Minnesota, 2000 through 2006. J Food Protect. 2010;73:2059–2064.
6. Irwin K, Ballard J, Grendon J, Kobayashi J. Results of routine restaurant inspections can predict outbreaks of foodborne illness: the Seattle-King County experience. Am J Public Health. 1989;79:586–590.
7. Harrison C, Jorder M, Stern H, et al. Using online reviews by restaurant patrons to identify unreported cases of foodborne illness—New York City, 2012-2013. MMWR Morb Mortal Wkly Rep. 2014;63:441–445.
8. Harris JK, Mansour R, Choucair B, Olson J, Nissen C, Bhatt J. Health department use of social media to identify foodborne illness—Chicago, Illinois, 2013-2014. MMWR Morb Mortal Wkly Rep. 2014;63:681–685.
9. Marziano S. Cutgroup #9—Foodborne Chicago. http://www.smartchicagocollaborative.org/cutgroup-9-foodborne-chicago
Updated 2014. Accessed May 24, 2016.
11. National Association of County & City Health Officials. National Profile of Local Health Departments. http://nacchoprofilestudy.org
. Updated 2014. Accessed July 10, 2016.
12. Veil SR, Buehner T, Palenchar MJ. A work‐in‐process literature review: incorporating social media in risk and crisis communication. J Contingencies Crisis Manage. 2011;19:110–122.
13. Song C, Lee J. Citizens' use of social media in government, perceived transparency, and trust in government. Public Perform Manage Rev. 2016;39:430–453.
14. Duggan M, Ellison NB, Lampe C, et al. A Social Media Update 2014. Washington, DC: Pew Research Center; 2015:1–17.
15. Hawkins JB, Brownstein JS, Tuli G, et al. Measuring patient-perceived quality of care in US hospitals using Twitter
. BMJ Qual Saf. 2016;25:404–413.
16. Ung J. ARKHumanity app saves lives one tweet at a time. The State Press. September 17, 2015. http://www.statepress.com/article/2015/09/ark-humanity-app-save-lives
. Accessed September 6, 2017.