DoD Looks for Disaster Response Solutions Through Artificial Intelligence Innovation Competition

John KoenigAn important new trend in the federal sector involves the use of "innovation competitions" to develop Artificial Intelligence (AI) solutions for major problems that the agencies face. An ongoing AI innovation competition from Centers for Medicare and Medicaid Services (CMS), known as the Artificial Intelligence (AI) Health Outcomes Challenge began in March 2019. While that competition is only open to major consulting firms, the Department of Defense (DoD) was the trendsetter here and has been working on a series of AI competitions that began with the xView competition in March 2018. Both the xView competition and the subsequent xView2 competition were open to the public, and I will be providing an in-depth review of the xView2 competition in this article.

In recent years, DoD has launched several units who each have been tasked with making sure that the US military has the most advanced AI tools at its disposal. In my last article - How DoD Plans to Leverage Artificial Intelligence and Open Source to Improve Emergency Response and Disaster Relief, I introduced the Joint Artificial Intelligence Center (JAIC) and the Defense Innovation Unit (DIU). The JAIC is the DoD's "Center of Excellence" for AI and the unit works to prepare the US military for AI by developing strategy, writing policy, and building relationships. DIU has a similar mission with a different focus. Instead of sitting inside the Pentagon, DIU is headquartered in Silicon Valley, where they work with the world's leading technology companies to bring commercial tech solutions into the military ecosystem.

Leading edge AI technologies are extremely powerful and require major investments to implement. Figuring out how to develop and procure these technologies is a serious problem that the JAIC and DIU have been trying to solve. The latest plan to sidestep the arduous federal procurement process and weed out federal contractors who can't get the job done is the xView2 AI image analysis innovation competition. Open from August - December 2019, the competition offered $150,000 in total prizes and an option for the winner to start their own company and sell their solution to the government.

The AI Challenge

When disaster strikes, speed is critical. The time it takes to properly assess damage in the wake of a major event can be the difference between life and death. However, emergency responders must often navigate disruptions to local communication and transportation infrastructure, making accurate assessments dangerous, difficult and slow. And while satellite and aerial imagery offer less risky alternatives that cover more ground, analysts must still conduct manual, time-intensive assessments of images.

Satellite images show a residential subdivision before and after the October 2017 Tubbs Fire in Santa Rosa, CA - Digital Globe/MAXARThe Defense Innovation Unit's xView2 Challenge sought to find ways to automate post-disaster damage assessment. DIU is challenging machine learning experts to develop computer vision algorithms that will speed up analysis of satellite and aerial imagery by localizing and categorizing various types of building damage caused by natural disasters.

The xView2 Challenge was DIU's second prize competition focused on furthering innovation in computer vision for humanitarian assistance and disaster relief efforts. The xView2 Challenge built upon the xView1 Challenge. xView1 sought computer vision algorithms to locate and identify distinct objects on the ground useful to first responders.

"DIU's goal in hosting this challenge is to enlist the global community of machine learning experts to tackle a critically hard problem: detecting key objects in overhead imagery in context and assessing damage in a disaster situation," said Mike Kaul, DIU AI portfolio director.

"We are always looking for ways to improve rapid damage assessment to ensure we and our partners deliver the right resources to the right places at the right time, and we are confident the DIU Challenge can contribute to that goal," said FEMA Regional Administrator Robert Fenton, a partner in the challenge.

DIU led a team of experts from academia and industry to create a new dataset, xBD, to enable localization and damage assessment before and after disasters. The dataset will provide the foundation for the challenge. While several open datasets for object detection from satellite imagery already exist - for example, SpaceNet and xView - each represent only a single snapshot in time and lack information about the type and severity of damage following a disaster.

This satellite image shows damaged (in red) and undamaged (light blue) buildings in the Santa Rosa residential subdivision pictured above following the Tubbs Fire - Digital Globe/MAXARThe largest and most diverse annotated building damage dataset, xBD allows Machine Learning and AI practitioners to generate and test models to help automate building damage assessment. The open source electro-optical imagery (0.3 m resolution) xBD dataset will encompass 700,000 building annotations across 5,000 square kilometers of freely available imagery from 15 countries. Seven disaster types are included: wildfire, landslides, dam collapses, volcanic eruptions, earthquakes/tsunamis and wind and flooding damage.

There were three competition prize tracks for the xView2 Challenge:

  1. Open source. Teams compete for leaderboard positions and awards for top scores. By releasing their models publicly under a permissive open-source license, teams also become eligible for an additional open-source award.
  2. Nonexclusive government purpose rights. Teams grant government purpose rights to become eligible for awards or top scores on the leaderboard. Solutions can be used to help future disaster recovery efforts.
  3. Evaluation Only. Teams retain their intellectual property and only grant DIU the right to benchmark their solution and compete for leaderboard position. Top teams in this category will still be eligible for a special monetary prize pool for their submissions.

The best solutions for all three categories were eligible for a share of a $150,000 prize purse. Top solvers were also invited to present their work at the December NeurIPS 2019 Workshop on AI for humanitarian assistance and disaster relief. Winners of any cash prize will be considered eligible to be awarded follow-on work with the Defense Department. The competition started in August month and ran through November.

Findings will be applied in both operational and academic use cases that include, but are not limited to:

  • Obstructed roads
  • Rerouting across obstructed roads
  • Force of nature identification
  • Resource allocation decision-making
  • Object recognition
  • Object identification

Baseline models, developed collaboratively between DIU and Carnegie Mellon's Software Engineering Institute, were publicly available as a starting point for the Challenge. In addition to advancing the state of the art in damage assessment, it is envisioned that the xBD dataset will provide researchers, companies and other groups with the means and motive to develop algorithms that bring humanitarian assistance and disaster response into the age of AI.