{"id":123010,"date":"2022-03-14T17:31:57","date_gmt":"2022-03-14T17:31:57","guid":{"rendered":"https:\/\/news.microsoft.com\/?p=445768"},"modified":"2022-03-14T17:31:57","modified_gmt":"2022-03-14T17:31:57","slug":"how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2022\/03\/14\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers\/","title":{"rendered":"How AI can support social interaction between children who are blind and their peers"},"content":{"rendered":"<figure class=\"wp-block-image alignwide size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"2560\" height=\"1441\" src=\"https:\/\/www.sickgaming.net\/blog\/wp-content\/uploads\/2022\/03\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers.jpg\" alt=\"A young boy wearing the PeopleLens sits on the floor of a playroom holding a blind tennis ball in his hands. His attention is directed toward a woman sitting on the floor in front of him holding her hands out. The PeopleLens looks like small goggles that sit on the forehead. The image is marked with visual annotations to indicate what the PeopleLens is seeing and what sounds are being heard.\" class=\"wp-image-825280\"><figcaption>The PeopleLens is a new research technology designed to help people who are blind or have low vision better understand their immediate social environments by locating and identifying people in the space. Coupled with a scheme of work based on research and practices from psychology and speech and language therapy, the system can help children and young people who are blind more easily forge social connections with their peers.<\/figcaption><\/figure>\n<p>For children born blind, social interaction can be particularly challenging. A child may have difficulty aiming their voice at the person they\u2019re talking to and put their head on their desk instead. Linguistically advanced young people may struggle with maintaining a topic of conversation, talking only about something of interest to them. Most noticeably, many children and young people who are blind struggle with engaging and befriending those in their age group despite a strong desire to do so. This is often deeply frustrating for the child or young person and can be equally so for their support network of family members and teachers who want to help them forge these important connections.<\/p>\n<div class=\"annotations\" data-bi-area=\"margin-callout\">\n<ul class=\"annotations__list annotations__list--left\">\n<li class=\"annotations__list-item\"> <span class=\"annotations__type\">PUBLICATION<\/span> <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/peoplelens\/\" data-bi-type=\"annotated-link\" aria-label=\"PeopleLens\" data-bi-area=\"margin-callout\" data-bi-cn=\"PeopleLens\"> PeopleLens <\/a> <span class=\"svg-icon icon-chevron-right-med-blue\"><\/span> <span class=\"annotations__caption\">The PeopleLens is an open-ended AI system that offers people who are blind or who have low vision further resources to make sense of and engage with their immediate social surroundings.<\/span> <\/li>\n<\/ul>\n<\/div>\n<p>The <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/peoplelens\/\" target=\"_blank\" rel=\"noreferrer noopener\">PeopleLens<\/a> is a new research technology that we\u2019ve created to help young people who are blind (referred to as <em>learners<\/em> in our work) and their peers interact more easily. A head-worn device, the PeopleLens reads aloud in spatialized audio the names of known individuals when the learner looks at them. That means the sound comes from the direction of the person, assisting the learner in understanding both the relative position and distance of their peers. The PeopleLens helps learners build a <em>People Map<\/em>, a mental map of those around them needed to effectively signal communicative intent. The technology, in turn, indicates to the learner\u2019s peers when the peers have been \u201cseen\u201d and can interact\u2014a replacement for the eye contact that usually initiates interaction between people.<\/p>\n<p>For children and young people who are blind, the PeopleLens is a way to find their friends; however, for teachers and parents, it\u2019s a way for these children and young people to develop competence and confidence in social interaction. An accompanying scheme of work aims to guide the development of spatial attention skills believed to underpin social interaction through a series of games that learners using the PeopleLens can play with peers. It also sets up situations in which learners can experience agency in social interaction. A child\u2019s realization that they can choose to initiate a conversation because they spot someone first or that they can stop a talkative brother from speaking by looking away is a powerful moment, motivating them to delve deeper into directing their own and others\u2019 attention.<\/p>\n<p>The PeopleLens is an advanced research prototype that works on Nreal Light augmented reality glasses tethered to a phone. While it\u2019s not available for purchase, we are recruiting learners in the United Kingdom aged 5 to 11 who have the support of a teacher to explore the technology as part of a multistage research study. For the study, led by the University of Bristol, learners will be asked to use the PeopleLens for a three-month period beginning in September 2022. For more information, visit the <a href=\"https:\/\/forms.office.com\/Pages\/ResponsePage.aspx?id=MH_ksn3NTkql2rGM8aQVG0YHUfpzEr5Dq7xK0TZFQ8ZUMlpIR1dSUTdSUFM4MkMyMDhVSktHSkc1Ui4u\" target=\"_blank\" rel=\"noreferrer noopener\">research study information page<\/a>.&nbsp;<\/p>\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><\/figure>\n<h2>Research foundation&nbsp;<\/h2>\n<p>The scheme of work, coauthored by collaborators <a href=\"https:\/\/www.gold.ac.uk\/psychology\/staff\/pring\/\" target=\"_blank\" rel=\"noreferrer noopener\">Professor Linda Pring<\/a> and <a href=\"https:\/\/www.linkedin.com\/in\/dr-vasiliki-kladouchou-a1b19661\/\">Dr. Vasiliki Kladouchou<\/a>, draws on research and practice from psychology and speech and language therapy in providing activities to do with the technology. The PeopleLens builds on the hypothesis that many social interaction difficulties for children who are blind stem from differences in the ways children with and without vision acquire fundamental attentional processes as babies and young children. For example, growing up, children with vision learn to internalize a joint visual dialogue of attention. A young child points at something in the sky, and the parent says, \u201cBird.\u201d Through these dialogues, young children learn how to direct the attention of others. However, there isn\u2019t enough research to understand how joint attention manifests in children who are blind. A review of the literature suggests that most research doesn\u2019t account for a missing sense and that research specific to visual impairment doesn\u2019t provide a framework for joint attention beyond the age of 3. We\u2019re carrying out research to better understand how the development of joint attention can be improved in early education and augmented with technology.<\/p>\n<h2>How does the PeopleLens work?&nbsp;<\/h2>\n<p>The PeopleLens is a sophisticated AI prototype system that is intended to provide people who are blind or have low vision with a better understanding of their immediate social environment. It uses a head-mounted augmented reality device in combination with four state-of-the-art computer vision algorithms to <em>continuously<\/em> locate, identify, track, and capture the gaze directions of people in the vicinity. It then presents this information to the wearer through spatialized audio\u2014sound that comes from the direction of the person. The real-time nature of the system gives a sense of immersion in the People Map.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.sickgaming.net\/blog\/wp-content\/uploads\/2022\/03\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers.png\"><img decoding=\"async\" loading=\"lazy\" width=\"1542\" height=\"1316\" src=\"https:\/\/www.sickgaming.net\/blog\/wp-content\/uploads\/2022\/03\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers.png\" alt=\"A graphic overview of the PeopleLens system describes its functionality and experience features with accompanying icons. \" class=\"wp-image-825295\"><\/a><figcaption>The PeopleLens helps the child wearing it build a mental map of those in their immediate social environment. Because the PeopleLens reads aloud the names of identified people in spatialized audio, the child is able to get a sense of the respective positions and distances of their peers. The system receives images and processes them with computer vision algorithms, as shown by the overlays on the top images in this screenshot of the PeopleLens development environment. The system then stiches together a world map that\u2019s used to drive the experiences, as shown at the bottom right.<\/figcaption><\/figure>\n<\/div>\n<p>The PeopleLens is a ground-breaking technology that has also been designed to protect privacy. Among the algorithms underpinning the system is facial recognition of people who\u2019ve been registered in the system. A person registers by taking several photographs of themselves with the phone attached to the PeopleLens. Photographs aren\u2019t stored, instead converted into a vector of numbers that represent a face. These differ from any vectors used in other systems, so recognition by the PeopleLens doesn\u2019t lead to recognition by any other system. No video or identifying information is captured by the system, ensuring that the images can\u2019t be maliciously used.<\/p>\n<p>The system employs a series of sounds to assist the wearer in placing people in the surrounding space: A percussive bump indicates when their gaze has crossed a person up to 10 meters away. The bump is followed by the person\u2019s name if the person is registered in the system, is within 4 meters of the wearer, and both the person\u2019s ears can be detected. The sound of woodblocks guides the wearer in finding and centering the face of a person the system has seen for 1 second but hasn\u2019t identified, changing in pitch to help the wearer adjust their gaze accordingly. (Those people who are unregistered are acknowledged with a click sound.) Gaze notification can alert the wearer to when they\u2019re being looked at.&nbsp;<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.sickgaming.net\/blog\/wp-content\/uploads\/2022\/03\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers-1.png\"><img decoding=\"async\" loading=\"lazy\" width=\"1400\" height=\"788\" src=\"https:\/\/www.sickgaming.net\/blog\/wp-content\/uploads\/2022\/03\/how-ai-can-support-social-interaction-between-children-who-are-blind-and-their-peers-1.png\" alt=\"A graphic overview of the PeopleLens system describes its functionality and experience features with accompanying icons.\" class=\"wp-image-825307\"><\/a><figcaption>The functionality of the PeopleLens system includes experience features such as recognizing a person in front of the wearer; attention notifications from the direction of those who look at the wearer; the ability to follow someone; and an orientation guide to help wearers find people and faces.<\/figcaption><\/figure>\n<\/div>\n<h2>Community collaboration<\/h2>\n<p>The success of the PeopleLens, as well as systems like it, is dependent on a prototyping process that includes close collaboration with the people it is intended to serve. Our work with children who are blind and their support systems has put us on a path toward building a tool that can have practical value and empower those using it. We encourage those interested in the PeopleLens to reach out about participating in our study and help us further evolve the technology.&nbsp;<\/p>\n<p><em>To learn more about the PeopleLens and its development, check out the <\/em><a href=\"https:\/\/news.microsoft.com\/innovation-stories\/project-tokyo\/\"><em>Innovation Stories blog<\/em><\/a><em> about the technology.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The PeopleLens is a new research technology designed to help people who are blind or have low vision better understand their immediate social environments by locating and identifying people in the space. Coupled with a scheme of work based on research and practices from psychology and speech and language therapy, the system can help children [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":123011,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[49],"tags":[50],"class_list":["post-123010","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-news","tag-recent-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/123010","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=123010"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/123010\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/123011"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=123010"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=123010"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=123010"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}