HomeNewsArticle Display

AFRL team takes top honors at international Artificial Intelligence competition

Large Scale Movie Description Challenge

Dr. Scott Clouse, Air Force Research Laboratory Decision Science Branch, Multi-Domain Sensing Autonomy Division, Sensors Directorate senior research engineer, said although there are significant improvements yet to be made on their work, with being able to take advantage of all the data and compute they have now, they’re on a path to making leaps and bounds in improvements in a lot of different facets of life not yet achieved. (Courtesy photo)

WRIGHT-PATTERSON AIR FORCE BASE, Ohio (AFNS) -- As part of an increased commitment to autonomy research, a team from the Air Force Research Laboratory at Wright-Patterson Air Force Base recently entered and won the Large-Scale Movie Description Challenge at the 2017 International Conference on Computer Vision in Venice, Italy.

“International open competitions such as the LSMDC provide an objective assessment of the latest state-of-the-art in cutting edge Artificial Intelligence technology,” said Dr. Vincent Velten, AFRL’s Multi-Domain Sensing Autonomy Division Decision Science Branch Technical advisor.

The goal of the LSMDC was to automatically generate a simple one sentence description of the actions or activities that occur in a 4-5 second video clip from a movie. Participants were given access to a training data set of clips and associated human generated sentences and were required to provide an algorithm for independent human evaluation against a blind test set of movie clips.

The AFRL team, comprised of Dr. Scott Clouse, senior research engineer at the Decision Science Branch; Oliver Nina, a PhD student from Ohio State University and also a research intern on a Defense Department Science, Mathematics and Research for Transformation, or SMART, Scholarship for Service fellowship at AFRL; and Nina’s advisor, Dr. Alper Yilmaz, also from OSU, were victorious over world leaders in Artificial Intelligence research such as Facebook AI research, the University of Toronto and Ecole Polytechnic de Montreal.

“This result prominently places the AFRL team in the AI research field and demonstrates an advanced technology that is a key enabling component of Air Force autonomy goals,” said Velten. “This technique can eventually be used to automate the screening of video streams to alert operators to operationally important events for systems such as Predator/Reaper and Global Hawk.”

Nina said humans, in the form of a three-judge panel, evaluated the submitted algorithms rather than computers as in previous years.

For people who are hearing or visually impaired, enjoying a commercial film sometimes requires additional support beyond the traditional format. That may be provided by some kind of accessibility to that media. One of the means of doing that are audio descriptive services that provide sort of an audio book version of the film so people can enjoy it, explained Clouse.

The goal of the LSMDC challenge was to produce a system that can turn such a film into this audio description format.

“Currently, they’re produced in kind of a theatrical way, just like the film is, where you have writers converting a script or a screenplay into more of a prose format,” said Clouse. “The reader then has to be skilled enough to convey the information in a more theatrical type of way. Then the movie dialogue plays along as it would normally, so they kind of have to interject as they go with the film. In addition to the dialogue, you want to describe what’s going on. That’s the point of descriptive services for people, who in particular, are visually impaired.”

The point is to generate these services in an automated way to cut down on the cost of generating the capability, explained Clouse.

“Because of the cost and time required to produce these kinds of descriptions, it is not easy to access them for a lot of different films and television shows,” said Clouse. “There are a very limited number of these available. It’s a very human intensive process to generate these materials so it’s the fundamental limitation of the throughput of people. If you can generate them automatically, then you’ve got a nice description as well as the audio that goes along with the film.”

The Air Force would like to similarly produce descriptions of video sequences that are captured from surveillance platforms or any kind of data feed, according to Clouse.

“Video is very popular with the sharp end of the Air Force because people very naturally deal with watching video and understanding what’s going on there,” added Velten. “However, there is a lot of it and not that many people to do the equivalent of this sort of function for a military application.”

Analysts may have to watch 50 hours of video to find 5 minutes of something interesting that’s militarily pertinent and matters for intelligence purposes or even for doing a special operations mission rehearsal, said Velten.

“This sort of technology would allow us to index clips, like you would in a library, and just show us the interesting parts. The nice thing is that there’s a civilian analogue to it and so there is a lot of great civilian research and the AFRL team showed they are at the forefront of that. The real motivation for the military is to be able to sort through enormous amounts of video and describe actions that are going on,” said Velten.

The team worked with 101,000 short video clips that were provided for this year’s competition. The algorithm they developed takes the video clips and produces a sort of abstract summary that is then translated into human-readable phrases.

“A great deal of computer crunching was required to do this and the team was able to use the super computer called Thunder at AFRL’s Defense Department Supercomputing Resource Center,” said Velten.

Thunder is part of the DoD High Performance Computing Modernization Program. Velten added that without Thunder, the research would not have been possible.

“We’re trying to mimic what the brain is doing,” said Nina. “We can help the blind, or the visually impaired. Together we can reach great goals to help humanity and to help the Air Force defend our country.”

Clouse indicated there are many significant improvements yet to be made, but with being able to take advantage of all the data and compute they have now, there’s a path to making leaps and bounds in improvements in a lot of different facets of life not yet achieved.

“The whole point is to produce systems that can have more human-like qualities in terms of their ability to not only produce output from fairly limited input, but also to produce output that human beings can trust. This is a very difficult problem.

“Obviously, this has enormous defense applications, but even larger societal and commercial applications. There are some potentially very impressive things right around the corner,” said Velten.


Facebook Twitter
#ICYMI Yesterday, #USAF Chief of Staff @GenDaveGoldfein had the privilege of presenting the Air Force Cross to Tech… https://t.co/E1f8TBgJYn
Planning to visit the National #911PentagonMemorial? Know before you go; the memorial will close temporarily later… https://t.co/8dBsvEl1gW
RT @ActingSecAF: Op-ed | The U.S. Space Force must be independent but not insular - https://t.co/VAoFe3oXF1 https://t.co/swaIMlVVDZ
👋 Hello @CityOfDallas, #AmericasAirForce is heading your way Sept. 27. Mark your calendars and meet us there!… https://t.co/MWnoLe8aSF
RT @DeptofDefense: Can we fix it?! 👷 $106,176 is how much money the @usairforce saved, thanks to Tech. Sgt. Keith Boudreau’s innovative 3D-…
RT @WomenInAviation: USAF women pilots to provide inspiration to future female aviators https://t.co/HsQuy6RKSH #WomeninAviation #GIAD19 #F
RT @AFResearchLab: "The technologies of tomorrow will exist because of the "Basic Research" of today." Read more on how we're collaboratin…
#WatchLive today at 9:00 A.M. EDT as Barbara M. Barrett, Secretary of the Air Force nominee, testifies before Congr… https://t.co/7DsAHZSZRL
RT @PACAF: #PACAF photo of the Week! #Defenders from bases around the world converged on #Guam to drill in close quarters urban combat as…
An Oral History of 9/11 - Commander Anthony Barnes, "That first hour was mass confusion because there was so much e… https://t.co/cqmFnxB9LA
RT @ActingSecAF: We will #NeverForget the lives lost, both victims and first responders, or those Airmen who have sacrificed so much since…
RT @ActingSecAF: Thanks @JohnBoozman & @RepGaramendi for hosting an early #USAF birthday celebration & honoring the service of our #Airmen
RT @DeptofDefense: WATCH LIVE: @POTUS Donald Trump, Secretary of Defense Dr. Mark T. Esper, & Chairman @thejointstaff @USMC Gen. Joe Dunfor…
RT @AirmanMagazine: The paint job on these @48FighterWing F-15s is similar to that of the P-47 Thunderbolt, the primary aircraft used by th…
Six years ago, Lt. Col. Dan Magruder lost his friend and fellow #AirForce Veteran to suicide. Their training didn't… https://t.co/fY7sVgrriF
#ICYMI, the #F22 turned 22! Happy Birthday! https://t.co/KH4m6FnJk5
RT @Eagles: Thanks to the 177th Fighter Wing for today's amazing flyover! #AFFlyover | #FlyEaglesFly https://t.co/s7EJqi1O1l
RT @AirmanMagazine: A B-2 Spirit from @Whiteman_AFB received fuel from a KC-135 Stratotanker above the Norwegian Sea, Sept. 5, 2019. This e…