Infectious disease surveillance 2.0: Crawling the Net to detect outbreaks

July 8, 2008 (Computerworld) While recent outbreaks of salmonella in the U.S. have made headlines, an automated real-time system that scours the Web for information about disease outbreaks spied early reports in New Mexico about suspicious gastrointestinal illnesses days before the U.S. Centers for Diseases Control and Prevention (CDC) issued an official report on the problem.

The system, called HealthMap, is a free data-mining tool that extracts, categorizes, filters and links 20,000 Web-based data sources such as news sites, blogs, e-mail lists and chat rooms to monitor emerging public health issues. HealthMap, which is profiled in the July issue of the journal Public Library of Science Medicine and is open to anyone, was developed in late 2006 by John Brownstein and Clark Freifeld. Both men work in the informatics program at Children's Hospital Boston.

The system's goal is to try to detect early outbreaks of diseases before they are spotted by traditional surveillance sources or the international public health community, Brownstein said. Often, reports of new occurrences of infectious disease surface on the Web well before they percolate to the attention of the existing public health reporting infrastructure, he said.

"There were media reports and chat room discussions -- things going on in the informal settings -- saying things about the SARS outbreak [in China] before traditional official channels were reporting on the outbreak," he noted.

As of July 8, the tool was tracking 226 reports of salmonella, 110 reports of avian influenza, 49 reports of Dengue fever, 28 reports of anthrax, 24 reports of West Nile virus and 21 cases of hemorrhagic fever worldwide.

The tool is especially vital to developing countries or other areas of the world that might not have traditional disease surveillance mechanisms in place, Brownstein said. "There was no real source that brought all information about outbreaks together," he said. "This is a way to bring all this information together in a very organized and synthesized way while filtering a lot of noise that might otherwise exist on the Web."

HealthMap scours Web sources every hour, seven days a week, Freifeld added. The pair use RSS feeds when available to access the data or screen-scraping techniques for Web sources not available in feeds. The data is then categorized by the type of disease being described and the location of the outbreak, Freifeld added.

Articles are then analyzed for duplication and content, with duplicate articles removed and those that include new information about a related topic added to a Google Map. On the map, alerts are color-coded for how much recent news about those outbreaks has been noted. HealthMap also includes pop-up windows on highlighted cities or states that provide links to all the news reports on an outbreak for that location.