Microsoft Launches App That Describes the World to the Blind Using AI

App users are able to point their phone camera at a person or thing and the app will identify them and provide information for the user. For example, if a user points the camera at a person, the app will say who they are and how they’re feeling. People the user hangs out with a lot can be saved to the app, meaning they are easily recognizable when the camera is pointed at them. The app can also identify a product and tell the user what it is.
Seeing AI uses neural networks to identify exactly what it’s looking at. Most of the functions are carried out on the device itself, which makes results much faster. But some more complex features like recognizing hand writing require a connection to the internet to get results via the cloud.

The app can also identify U.S. currency, read text, and reads product bar codes. The app also directs the user how to get the camera in the correct position to identify what is in front of it.

“One of the things we wanted to do was face recognition on device, and we’ve done that so within a few milliseconds you’ll hear the result. It’s all about the speed, and we try to do as much as we can on the device,” Saqib Shaikh, the tech lead on Seeing AI told The Verge.

Shaikh, who lost his sight when he was 7, announced the project in a video in March 2016, saying he had dreamed of building something that at any moment could tell you what is going on around you since he was at university.

The app was launched today, but is only available on iOS at this time.