Human eyes assist drones, teach machines to see

Who would win in a real-life game of “Where’s Waldo,” humans or computers? A recent study suggests that when speed and accuracy are critical, an approach combing both human and machine intelligence would take the prize. With drones being used to monitor everything natural disaster sites, pollution, or wildlife populations, analyzing drone images in real-time has become a critically important big data challenge. Publishing in the journal Big Data, researchers, including Stéphane Joost from EPFL, present a new approach to rapidly interpret aerial images taken by camera drones that combines human crowdsourcing and machine learning.

To develop and test their approach, the researchers traveled to the Kuzikus wildlife reserve in the heart of Namibia. There they surveyed the compound using camera-drones to count the resident wildlife population, including ostriches, kudus, wildebeests, zebras and rhinos. Confronted with the daunting number of images captured, and the difficulty of having computers reliably distinguish between rocks, bushes, and animals, the researchers turned to the internet. Using an online crowdsourcing platform, www.micromappers.org, they uploaded their images for manual analysis by an army of digital volunteers.

The task put before the digital volunteers was straightforward. They were asked to click through a stack of images, identify all animals, and outline them on their screens. According to the study’s senior author Stéphane Joost, the response was astounding: “Within two days, they had evaluated 98% of the 26,000 images that had been uploaded.” Next, half of these annotated images were used to train an automatic object recognition algorithm, which was then tested on the remaining images. “The 500 digital volunteers did generate a number false positives, tracing features that in actual fact were not animals. Despite that, their analysis was certainly good enough to serve as training data for the computer algorithm,” says Joost.

Essential aid following disasters
Artificial intelligence has already become a mainstay in disaster response operations, where it is primarily relied on to scour tweets and text messages for what can be life-saving information. “Twitter can erupt with up to tens of thousands of tweets sent out of the disaster zone per minute. It would take humans days to sort the relevant from the irrelevant information,” says Joost. “Today, this challenge has been solved and tweets can be analyzed quickly using a combination human and artificial intelligence. Our challenge was to transpose this solution to apply it to the analysis of image data.”

From high above, drones carrying high-resolution cameras offer rescue workers an invaluable vantage point from which to locate damaged buildings, blocked roads, and other important features. And fortunately, automatically spotting and identifying animals in the Savanna is likely to be far more challenging than locating infrastructure destroyed in a disaster. Following the completion of the pilot study in Namibia, the new tool was used in its first serious mission: it was adopted by the World Bank to speed up damage and needs assessment in the aftermath of tropical cyclone Pam that hit the Island State of Vanuatu in 2015.

An exceptional list of authors
“It is interesting to point out that three of the study’s co-authors were, at the time, environmental engineering students, says Joost. It is quite rare for students to become co-authors of scientific research articles. “Nicolas Rey, Julien Briant, and Pauline Millet were attending a course in territorial imagery at EPFL, where they developed some of the detection algorithms used in this project.“

The study was carried out by researchers from EPFL, the Qatar Computing Research Institute, the Kuzikus Wildlife Reserve in Namibia, and the University of Zurich, in collaboration with Drone Adventures.