Merlin Bird Photo ID Mobile App Launches

Computer vision app can identify North American bird species from photographs

Ithaca, NY, New York, NY, & Pasadena, CA — The Merlin Bird Photo ID mobile app has been launched and, thanks to machine-learning technology, can identify hundreds of North American species it “sees” in photos. The app was developed by Caltech and Cornell Tech computer vision researchers in partnership with the Cornell Lab of Ornithology and bird enthusiasts. Because Merlin Bird Photo ID can be used on mobile devices, it can go anywhere bird watchers go.

“When you open the Merlin Bird Photo ID app, you’re asked if you want to take a picture with your smartphone or pull in an image from your digital camera,” explains Merlin project leader Jessie Barry at the Cornell Lab. “You zoom in on the bird, confirm the date and location, and Merlin will show you the top choices for a match from among the 650 North American species it knows.”

Caltech and Cornell Tech computer scientists trained Merlin to recognize birds by showing it nearly one million photos that were collected and annotated by birders and volunteers mobilized by the Cornell Lab. Merlin scans its photo database for possible matches. Then, like any good birder, the system considers species that would be found at that specific time of year and in that location using information from the eBird program which collects an average of seven million bird observation records each month from around the world.

“In building Merlin Bird Photo ID we were especially concerned with the quality and the organization of the data,” says Serge Belongie, professor of computer science at Cornell Tech. Together with professor Pietro Perona of Caltech, he is the co-founder of Visipedia, the Google-funded umbrella project that is using advances in machine learning and computer vision to help classify objects in photographs. “Ultimately we want to create an open platform that any community can use to make a visual classification tool for butterflies, frogs, plants, or whatever they need.”

“This app is the culmination of seven years of our students’ hard work and is propelled by the tremendous progress that computer vision and machine learning scientists are making around the world. All of a sudden, our smartphones can really see!’’ says Perona, the Allen E. Puckett Professor of Electrical Engineering in the Caltech Division of Engineering and Applied Science. “This was a distant dream when I was a graduate student and now it’s finally happening.”

How good is Merlin Bird Photo ID? “Accuracy is around 90 percent if the user’s photo is of good quality. Submit a fuzzy image or one in which the bird is small or partially covered by leaves and the odds of getting an accurate match go down,”’ says Caltech postdoctoral researcher Steve Branson.

Despite the high-tech advances, humans are still an important part of the process. “We need eBird data from bird watchers along with experts who can label the photos used to train Merlin,” says Caltech graduate student Grant Van Horn. “You need teachers to teach the machine what it needs to do. Our system combines the expertise of hundreds of birders and ornithologists.” Van Horn and Branson are both part of the Visipedia team, and developed the algorithms that allow Merlin to learn to recognize the birds.

There are more thrills to come. Just around the corner is a Merlin Bird Photo ID release in Spanish for birds in Mexico. Down the road, the Merlin team will produce versions for South America, Europe, Asia, Africa, Australia—all parts of the world.

“The wonderful thing about this project is the collaboration with the Visipedia team,” says Barry. “We have a product that really works because it’s supported by fantastic research and is great for the birding community because it’s built for birders by birders. You just have to try it!”

Photo ID may be downloaded free for iOS or Android systems from the Apple and Google Play app stores. It is now included in the Merlin Bird ID app, which was originally released in 2014 to identify birds by asking users five questions about the birds they saw.

Merlin Photo ID is powered by Visipedia with support from Google, the Jacobs Technion-Cornell Institute, and the National Science Foundation.