Smartphone app helps the visually impaired ‘see’

Artificial intelligence can now help the visually impaired in their day-to-day lives.

Aipoly, an app created by an American startup born out of the Singularity University in California, helps the blind, visually impaired and color blind recognize their surroundings via a smartphone. All users have to do is take a picture of an object in front of them, which is then analyzed and tagged by the app before being read back to them through text-to-speech technology. Alberto Rizzoli, Aipoly’s co-founder, explains how the app is letting people ‘see’ the world around them.

How did the Aipoly project idea emerge?

It was born out of the Singularity University at the NASA Ames Research Park, where we were asked to create a startup that would eventually impact the lives of one billion people within 10 years, and do so using technologies that has become better, cheaper, and faster at an exponential rate. So we decided to use artificial intelligence and computer vision, and use it to impact a good cause. Recently, researchers in computer science have been able to make computers understand pictures and describe them, almost like a person would. Artificial intelligence has evolved to a point where it is reaching human intelligence in terms of object recognitionand we want to make use of that.

How does Aipoly empower the visually impaired to explore and understand their surroundings?

If you are visually impaired, you can lose a sense of context and exploration becomes much harder to do. Aipoly currently helps by identifying objects in real-time, like a talking camera. It’s fast enough that people can hold their phone, move their arm, and listen to it speak as it describes what it sees. It can be used to find sockets for charging a phone, to explore new areas, and even in public bathrooms, so that individuals avoid touching things.

Could you explain how the app works?

The app is a container for artificial intelligence (AI). This AI has one very special property.It can recognize things in real-time, like a human, even if it has never seen them before. For example, Aipoly can recognize the cup I am drinking from and distinguish it from the thousands of items it learned, but in reality, it has never seen this single cup design before. Thanks to a field of machine learning called “deep learning”, it was able to understand the concept of a cup, chair, table, and so on, for thousands of different items. The secret behind this is something called a convolutional neural network, which is an architecture of algorithms inspired by the animal visual cortex. Using powerful computers, we show the AI millions and millions of images with descriptions and teach it. As a result, we obtain a neural network trained on those images, which represent its view of reality.

What has been the reaction of the visually impaired to the app?

We receive emails daily from users who are excited and thankful for the technology. Some tell us that they cried, others tell us of how it helped them in specific episodes. It’s very rewarding. There is still a long way to go before the technology becomes a must-have for the visually impaired. Nobody here thought this was going to be easy. So far, we’ve had an average of five star reviews and lots of happy users, but we know the best is yet to come and currently the app is only available in beta.

What other benefits could this app offer in the future?

In the future, Aipoly will recognize thousands more items and be useful in supermarkets, stores, and even in navigation as it recognizes specific brands, products, and elements of the home and outdoors like pavements, steps, and more.

How can users help Aipoly to recognize new objects?

Any sighted user can download the app.It’s free and helps a good cause.Just search for “Aipoly Vision”. There is a pencil-shaped button that once pressed, takes a picture which can be captioned. Once 1,000 users take a picture of the same item, for example a pineapple, the AI will learn what it is. So far, users from around the world have been teaching Aipoly items daily. We hope to grow this in the future so it can be more fun and social.

– ByDaniel Casillas