Firefighters have only their wits and five senses to rely on inside a burning building. But research developed in part by NASA's Jet Propulsion Laboratory, Pasadena, California, may change that, introducing artificial intelligence (AI) that could collect data on temperatures, gases and other danger signals and guide a team of first responders safely through the flames.

AUDREY, the Assistant for Understanding Data through Reasoning, Extraction, and sYnthesis, has received the Undersecretary's Award for Collaboration from the Department of Homeland Security (DHS) in recognition of its joint development by JPL and DHS. It's part of the Next Generation First Responder (NGFR) program, a DHS initiative to innovate new ways to keep firefighters, police, paramedics and other first responders safe in the field through increased awareness of their surroundings and communication abilities.

But the big picture is even more exciting: AUDREY can track an entire team of firefighters, sending relevant signals to individuals while helping to make recommendations for how they could work together.

"As a firefighter moves through an environment, AUDREY could send alerts through a mobile device or head-mounted display," said Mark James of JPL, lead scientist for the AUDREY project.

AUDREY is designed to be integrated with the "Internet of Things" - the idea of numerous devices and sensors all wirelessly "talking" to one another. In the case of firefighters, wearable sensors in their clothes could pick up their GPS location, heat in other rooms, the presence of dangerous chemicals and gases, satellite imagery of a location and much more.

"When first responders are connected to all these sensors, the AUDREY agent becomes their guardian angel," said Edward Chow, manager of JPL's Civil Program Office and program manager for AUDREY. "Because of all this data the sensor sees, firefighters won't run into the next room where the floor will collapse."

John Merrill, NGFR program manager for the DHS Science and Technology Directorate, said that technology is rapidly providing new strengths for first responders in the field.

"The proliferation of miniaturized sensors and Internet of Things devices can make a tremendous impact on first responder safety, connectivity, and situational awareness," Merrill said. "The massive amount of data available to the first responder is incomprehensible in its raw state and must be synthesized into useable, actionable information."

Guardian angel in the cloud
AUDREY is designed to keep watch from above. As a cloud-based piece of software, it can do more than send data to those in the field. As it watches an event, it can actually learn and start making predictions about what resources will be needed next.

James said the system is designed to recognize the specific roles of first responders in the field. This allows AUDREY to provide potentially lifesaving information customized to the various roles, which avoids overloading the users.

"Since AUDREY knows the roles of everyone who receives her data, she only supplies the relevant information that is appropriate for them," James said.

In June, AUDREY was tested in a virtual demonstration at the Public Safety Broadband Stakeholder Meeting held by the Department of Commerce in San Diego. It was fed data from a variety of sensors and asked to make safety recommendations, which it then sent to a mobile device. Within a year, Chow said, the plan is to test AUDREY in field demonstrations.

From machine to human-like thinking
Chow emphasized that artificial intelligence is only as effective as the data it's working with. The more data it has, the higher the probability that it will make useful recommendations.

"Most A.I. projects are rule-based - if this, then that," he said. "But what if you're only getting part of the information? We use complex reasoning to simulate how humans think. That allows us to provide more useful info to firefighters than a traditional A.I. system."

AUDREY includes several technologies developed by NASA and the Department of Defense over the last five years. AUDREY for NGFR has been in development for nine months and is funded by the Department of Homeland Security. Caltech manages JPL for NASA.