{"id":108212,"date":"2020-01-28T14:05:06","date_gmt":"2020-01-28T14:05:06","guid":{"rendered":"https:\/\/news.microsoft.com\/?p=435999"},"modified":"2020-01-28T14:05:06","modified_gmt":"2020-01-28T14:05:06","slug":"using-ai-people-who-are-blind-are-able-to-find-familiar-faces-in-a-room","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2020\/01\/28\/using-ai-people-who-are-blind-are-able-to-find-familiar-faces-in-a-room\/","title":{"rendered":"Using AI, people who are blind are able to find familiar faces in a room"},"content":{"rendered":"

Paralympics in Brazil<\/strong><\/h2>\n

Project Tokyo was born out of a challenge, in early 2016, from senior leaders at Microsoft to create AI systems that would go beyond completing tasks such as fetching sports scores and weather forecasts or identifying objects. Morrison said creating tools for people who are blind and with low vision was a natural fit for the project, because people with disabilities are often early adopters of new technology.<\/p>\n

\u201cIt is not about saying, \u2018Let\u2019s build something for blind people,\u2019\u201d Morrison said. \u201cWe are working with blind people to help us imagine the future, and that future is about new experiences with AI.\u201d<\/p>\n

Morrison and her colleague Ed Cutrell<\/a>, a senior principal researcher at Microsoft\u2019s research lab in Redmond, Washington, were tapped to lead the project. Both have expertise in designing technologies with people who are blind or with low vision and decided to begin by trying to understand how an agent technology could augment, or extend, the capabilities of these users.<\/p>\n

To start, they followed a group of athletes and spectators with varying levels of vision on a trip from the United Kingdom to the 2016 Paralympic Games in Rio de Janeiro, Brazil, observing how they interacted with other people as they navigated airports, attended sporting venues and went sightseeing, among other activities. A key learning, noted Cutrell, was how an enriched understanding of social context could help people who are blind or with low vision make sense of their environment.<\/p>\n

\u201cWe, as humans, have this very, very nuanced and elaborate sense of social understanding of how to interact with people \u2013 getting a sense of who is in the room, what are they doing, what is their relationship to me, how do I understand if they are relevant for me or not,\u201d he said. \u201cAnd for blind people a lot of the cues that we take for granted just go away.\u201d<\/p>\n

This understanding spurred a series of workshops with the blind and low vision community that were focused on potential technologies that could provide such an experience. Peter Bosher, an audio engineer in his mid-50s who has been blind most of his life and worked with the Project Tokyo team, said the concept of a technology that provided information about the people around him resonated immediately.<\/p>\n

\u201cWhenever I am in a situation with more than two or three people, especially if I don\u2019t know some of them, it becomes exponentially more difficult to deal with because people use more and more eye contact and body language to signal that they want to talk to such-and-such a person, that they want to speak now,\u201d he said. \u201cIt is really very difficult as a blind person.\u201d<\/p>\n

\"Microsoft
Ed Cutrell, a senior principal researcher with Microsoft\u2019s research organization in Redmond, Washington, is a co-leader of Project Tokyo. On his desk are several modified Microsoft HoloLenses that project researchers are using to help people who are blind and with low vision learn who is where in their social environment. Photo by Dan DeLong.<\/figcaption><\/figure>\n

A modified HoloLens<\/strong><\/h2>\n

Once the Project Tokyo researchers understood the type of AI experience they wanted to create, they set out to build the enabling technology. They started with the original Microsoft HoloLens<\/a>, a mixed reality headset that projects holograms into the real world that users can manipulate.<\/p>\n

\u201cHoloLens gives us a ton of what we need to build a real time AI agent that can communicate the social environment,\u201d said Grayson during a demonstration of the technology at Microsoft\u2019s research lab in Cambridge.<\/p>\n

For example, the device has an array of grayscale cameras that provide a near 180-degree view of the environment and a high-resolution color camera for high-accuracy facial recognition. In addition, the speakers above the user\u2019s ears allow for spatialized audio \u2013 the creation of sounds that seem to be coming from specific locations around the user.<\/p>\n

Machine learning experts on the Project Tokyo team then developed computer vision algorithms that provide varying levels of information about who is where in the user\u2019s environment. The models run on graphical processing units, known as GPUs, that are housed in the black chest that Grayson carted off to Regan\u2019s house for the user testing with Theo.<\/p>\n

One model, for example, detects the pose of people in the environment, which provides a sense of where and how far away people are from the user. Another analyzes the stream of photos from the high-resolution camera to recognize people and determine if they have opted to make their names known to the system. All this information is relayed to the user through audio cues.<\/p>\n

For example, if the device detects a person one meter away on the user\u2019s left side, the system will play a click that sounds like it is coming from one meter away on the left. If the system recognizes the person\u2019s face, it will play a bump sound, and if that person is also known to the system, it will announce their name.<\/p>\n

When the user only hears a click but wants to know who the person is, a second layer of sound that resembles an elastic band stretching guides the user\u2019s gaze toward the person\u2019s face. When the lens\u2019 central camera connects with the person\u2019s nose, the user hears a high-pitched click and, if the person is known to the system, their name.<\/p>\n

\u201cI particularly like the thing that gives you the angle of gaze because I\u2019m never really sure what is the sensible angle for your head to be at,\u201d said Bosher, who worked with the Project Tokyo team on the audio experience early in the design process and returned to the Cambridge lab to discuss his experience and check out the latest iteration. \u201cThat would be a great tool for learning body language.\u201d<\/p>\n

\"Peter
Peter Bosher, middle, an audio engineer who is blind who worked with the Project Tokyo team early in the design process, checks out the latest iteration of the system at Microsoft\u2019s research lab in Cambridge, UK, with researchers Martin Grayson, left, and Cecily Morrison, right. Photo by Jonathan Banks.<\/figcaption><\/figure>\n

Prototyping with adults<\/strong><\/h2>\n

As the Project Tokyo team has developed and evolved the technology, the researchers routinely invite adults who are blind or with low vision to test the system and provide feedback. To facilitate more direct social interaction, for example, the team removed the lenses from the front of the HoloLens.<\/p>\n

Several users expressed a desire to unobtrusively get the information collected by the system without constantly turning their heads, which felt socially awkward. The feedback prompted the Project Tokyo team to work on features that help users quickly learn who is around them by, for example, asking for an overview and getting a spatial readout of all the names of people who have given permission to be recognized by the system.<\/p>\n

Another experimental feature alerts the user with a spatialized chime when someone is looking at them, because people with typical vision often establish eye contact to initiate a conversation. Unlike the bump, however, the chime is not followed by a name.<\/p>\n

\u201cWe already use the name when you look at somebody,\u201d Grayson explained to Emily, a tester in her 20s who has low vision and visited the Cambridge lab to learn about the most recent features. \u201cBut also, by not giving the name, it might draw your attention to turn to somebody who is trying to get your attention. And by turning to them, you find out their name.\u201d<\/p>\n

\u201cI totally agree with that. That is how sighted people react. They capture someone out of the corner of their eye, or you get that sense, and go, \u2018Cecily,\u2019\u201d Emily said.<\/p>\n

The modified HoloLens the researchers showed to Emily also included an LED strip affixed above the band of cameras. A white light tracks the person closest to the user and turns green when the person has been identified to the user. The feature lets communication partners or bystanders know they\u2019ve been seen, making it more natural to initiate a conversation.<\/p>\n

The LED strip also provides people an opportunity to move out of the device\u2019s field of view and not be seen, if they so choose. \u201cWhen you know you are about to be seen, you can also decide not to be seen,\u201d noted Morrison. \u201cIf you know when you are being seen, you know when you are not being seen.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Paralympics in Brazil Project Tokyo was born out of a challenge, in early 2016, from senior leaders at Microsoft to create AI systems that would go beyond completing tasks such as fetching sports scores and weather forecasts or identifying objects. Morrison said creating tools for people who are blind and with low vision was a […]<\/p>\n","protected":false},"author":2,"featured_media":108213,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[49],"tags":[135,50],"class_list":["post-108212","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-news","tag-artificial-intelligence","tag-recent-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/108212","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=108212"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/108212\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/108213"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=108212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=108212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=108212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}