Visual Media Reasoning Program Gleans Intel from Cyberspace

By Amaani Lyle

American Forces Press Service

WASHINGTON, Jan. 10, 2014 – Software that can extract information from digital photos and videos that adversaries might post has made “finding a needle in a haystack” much easier, an analyst said during the Defense Advanced Research Projects Agency’s Congressional Tech Showcase here yesterday.

Mike Geertsen, DARPA program manager for the visual media reasoning program, said he and his team took state-of-the-art computer vision technology from around the world, synchronized it and added “DARPA reasoning” that allows them to search for valuable intelligence in unprecedented ways.

“[It’s] about unlocking the hidden intelligence in the photos and videos that we capture from our enemies,” he said. “If you can imagine going through every video, frame by frame, every photo, … we need some automated tools to help our warfighters and analysts know what the content of this critical media is.”

In the May 2011 Osama bin Laden raid, for example, analysts recovered scores of thumb drives, cameras and laptops with thousands of images on them – some of which might be valuable, Geertsen said.

The task to extract and analyze vital information in that incident alone could be daunting. But visual search reasoning filters the DARPA team has developed can scan thousands of images in seconds and can include weapons, facial recognition, vehicles, and even logos.

“We do this by fusing this technology and really understanding in detail the contents of every pixel in that video frame or image,” Geertsen said.

The program provides three different methods to conduct the search: natural clustering, query by example and content search, Geertsen said.

So if analysts seek, for example, images of an al-Qaida logo among millions of images, multi-size “bubbles” show what the system automatically bundles together as similar images, and the size of the clusters indicates the number of images in them.

“‘Similar’ might be patterns, colors, vehicles, anything,” Geertsen said. “We’re searching the contents of these photos in the wild, … so we add a little bit of common-sense reasoning to this system, and we’re able to get these kinds of results.”

Similarly, documents with key phrases now yield information based on some 65 different algorithms from various universities, all in communication with each other for the first time.

“Within seconds, I can look at 100,000 images, find 186 that have documents, and click on one to see why it’s similar,” Geertsen said. “It’s another quick way we’re creating for analysts and warfighters to visually search and explore massive amounts of media.”

The nascent program still has another year or so left in development, but law enforcement, the National Media Exploitation Center, U.S. Southern Command and others agencies are anxious for the program to be deployed, Geertsen noted.

“We’re getting inquiries from all sorts of people, and DARPA doesn’t own anything long-term. … We create the magic and hand it off to our federal partners.”