{"id":124608,"date":"2022-05-05T15:03:39","date_gmt":"2022-05-05T15:03:39","guid":{"rendered":"https:\/\/news.microsoft.com\/?p=446179"},"modified":"2022-05-05T15:03:39","modified_gmt":"2022-05-05T15:03:39","slug":"with-their-hololens-2-project-microsoft-and-volkswagen-collaborate-to-put-augmented-reality-glasses-in-motion","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2022\/05\/05\/with-their-hololens-2-project-microsoft-and-volkswagen-collaborate-to-put-augmented-reality-glasses-in-motion\/","title":{"rendered":"With their HoloLens 2 project, Microsoft and Volkswagen collaborate to put augmented reality glasses in motion"},"content":{"rendered":"

Imagine putting on a pair of augmented reality glasses as you head out in the morning and get into your vehicle. As your self-driving car takes you to your destination, you get holographic displays of traffic information, weather conditions, shopping recommendations and architectural highlights along the way. And as the day dawns, you can use holographic controls in front of you to adjust the interior temperature to your liking.<\/p>\n

That\u2019s the future of mobility envisioned by researchers at German automobile manufacturer Volkswagen, who see augmented reality as one of the key components of future mobility concepts. To get a little closer to that vision, Volkswagen collaborated with Microsoft to enable the HoloLens 2 mixed reality headset to be used in moving vehicles for the first time.<\/p>\n

The new \u201cmoving platform\u201d mode for HoloLens 2<\/a> overcomes a major limitation of mixed reality headsets and creates potential for the technology to be used in new ways \u2014 training drivers to handle challenging road conditions, for example, or creating new user experiences for autonomous vehicles. And while mobility is Volkswagen\u2019s focus, the capability could in future be shared across other industries.<\/p>\n

\"Black
Dr. Andro Kleen.<\/figcaption><\/figure>\n

\u201cWe think mixed reality information is the most intuitive information we could provide to enhance our customers\u2019 user experience,\u201d says Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation. \u201cBecause what you see there, and what you need to process, is very close to what humans normally see and process. It\u2019s not so abstract.\u201d<\/p>\n

An early adopter of augmented reality technology, Volkswagen introduced an augmented reality head-up display<\/a> in its ID. family of electric cars in 2020 that projects navigation arrows, lane markings and other information onto the environment in front of the cars.<\/p>\n

But Volkswagen was thinking about the potential of augmented reality even earlier, Kleen says. In 2015, for instance, Volkswagen embarked on a research project using self-piloted vehicles and augmented reality to teach driving on a racing circuit. Tested at Volkswagen\u2019s track facility in Ehra-Lessien, Germany, the Race Trainer<\/a> system used a head-up display that superimposed lines of arrows on the track for drivers to follow, and provided steering and braking cues to guide them through a gradated set of lessons.<\/p>\n

Volkswagen hoped to use HoloLens for the research project, but soon ran up against a problem. When the device was put into a moving vehicle, its sensors lost tracking and the holograms<\/a> it normally displays disappeared. Kleen\u2019s team approached Microsoft for help and connected with Marc Pollefeys, Microsoft director of science and an expert in 3D computer vision and machine learning. The Volkswagen team traveled to Microsoft\u2019s headquarters in Redmond, Washington to show Pollefeys and his team its work with HoloLens and its vision for augmented reality technology.<\/p>\n

\u201cWe had extensive discussions,\u201d says Pollefeys, now the director of Microsoft\u2019s Mixed Reality and AI Lab in Zurich, Switzerland. \u201cThey presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.\u201d<\/p>\n

The two teams began collaborating around 2018 to develop the moving platform feature for HoloLens 2, which required solving a fundamental problem. HoloLens uses two main types of sensors that measure its motion \u2014 visible light cameras and an inertial measurement unit, or IMU, that gauges acceleration and rotation speed. Together, the sensors mimic how humans see and move through the world.<\/p>\n

But similar to how being in a car or boat can cause motion sickness when what appears to be a stable environment is actually moving, when the tightly coupled HoloLens sensors start disagreeing in a moving environment \u2014 with the inertial measurement unit recognizing motion and the cameras unable to \u2014 things fall apart.<\/p>\n

\"Photo
Connecting inside and outside of the vehicle: A navigation element positioned outside the vehicle complements the 3D map integrated into the dashboard.<\/figcaption><\/figure>\n

To address that issue, Pollefeys\u2019 team developed an algorithm that models the discrepancies between the sensors and allows HoloLens to continue tracking. But testing the capability posed another challenge. Since Covid-19 restrictions in Switzerland made it impossible to try out the technology in a vehicle with another person at the time, Joshua Elsdon, a Microsoft senior software engineer who worked on the project, had to find solutions from his Zurich apartment.<\/p>\n

Elsdon devised a mock platform using a plastic box, sticking bits of tape inside to add visual texture and give the HoloLens cameras elements to track. He rode trams and buses around Zurich wearing a HoloLens headset, making sure its holograms held up as the vehicles moved. At night, Elsdon rode up and down elevators in his apartment block, testing the technology.<\/p>\n

\u201cWe had to do a lot of testing in my apartment. These aren\u2019t ideal development conditions,\u201d says Elsdon, who is now based in Redmond. \u201cAll of this stuff was done remotely and distributed across different countries, which was interesting.\u201d<\/p>\n

After the initial prototyping was done, the team also conducted testing on Puget Sound, near Microsoft\u2019s Redmond campus. They rented recreational boats, took them out on the water and used external measurement equipment to assess the performance of HoloLens\u2019 head-tracking system in a moving platform.<\/p>\n

Microsoft later tested the feature with Volkswagen. The Volkswagen researchers established a bidirectional data connection between the vehicle and the HoloLens in order to display and control real-time information from the car. Finally, the team implemented several demo use cases investigating how virtual interfaces could enhance the interior of future vehicles.<\/p>\n

\"Photo
The moving platform mode, combined with vehicle position data, enables HoloLens 2 to be used in new ways.<\/figcaption><\/figure>\n

\u201cWe connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car. This opens up completely new possibilities to not only display holograms within the driver\u2019s forward-facing field of view, but also wherever the user wearing the glasses is looking,\u201d says Michael Wittk\u00e4mper, augmented reality expert at Volkswagen.<\/p>\n

Microsoft rolled out the moving platform feature a few months ago and it is already attracting interest from maritime companies and organizations, which have been using HoloLens to remotely connect maritime workers with mechanical experts through Microsoft Dynamics 365 Remote Assist<\/a>. The app allows an expert in another location to look through the other person\u2019s HoloLens 2, share their field of view, diagnose a problem and provide input.<\/p>\n

Previously only usable when a ship is in port, the capability is even more needed when vessels are at sea and a piece of equipment breaks down.<\/p>\n

\u201cThe more remote the equipment or machine is, the harder it is to get the expert on site,\u201d says Pollefeys, who is also a professor of computer science at ETH Zurich, a public research university. \u201cThis feature turned out to be critical to unlock HoloLens 2 for the maritime space.\u201d<\/p>\n

HoloLens\u2019 moving platform feature is currently supported for use on large ships, and Microsoft plans to further refine it for use in elevators, trains, cars and other moving environments. Kleen envisions multiple ways the technology could be used to connect the inside of vehicles with the outside world, such as helping truck or bus drivers navigate through narrow streets, identifying points of interest along driving routes or providing entertainment for passengers.<\/p>\n

\"A
The direct communication of the AR headset with the vehicle data makes it possible to adjust the temperature or control the intensity and direction of the airflow by gesture.<\/figcaption><\/figure>\n

Kleen and Pollefeys believe augmented reality will become an increasingly important aspect of mobility in the future as smaller, more compact versions of smart glasses become available. Pollefeys characterizes mixed reality glasses as the third generation of personal computing devices, after personal computers and mobile phones.<\/p>\n

\u201cMicrosoft is not only interested in commercial devices like HoloLens, but also in the longer term, in devices that would make sense for consumers to use in daily life,\u201d Pollefeys says. \u201cWith augmented reality glasses, you could walk around the world and information can appear in context, where it\u2019s relevant. You could communicate with other people and also in 3D, as opposed to on a small screen.\u201d<\/p>\n

Kleen imagines augmented reality enabling seamless and connected mobility experiences, with people donning smart glasses as they leave home and receiving information through them, from navigational assistance to entertainment, as they travel through their day.<\/p>\n

\u201cWe think of this as moving toward a mobility system where different products and mobility solutions will be connected,\u201d Kleen says. \u201cThe basic assumption is that this technology will become lighter and smaller, and we think that as that happens, more people will get their hands on it and integrate it into their daily lives \u2014 and thus into their way of moving from A to B.\u201d<\/p>\n

Top photo: Volkswagen demonstrated the future of fully autonomous driving in its Gran Turismo concept car, the ID.VIZZION. (Images courtesy of Volkswagen)<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Imagine putting on a pair of augmented reality glasses as you head out in the morning and get into your vehicle. As your self-driving car takes you to your destination, you get holographic displays of traffic information, weather conditions, shopping recommendations and architectural highlights along the way. And as the day dawns, you can use […]<\/p>\n","protected":false},"author":2,"featured_media":124609,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[49],"tags":[1176,50],"class_list":["post-124608","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-news","tag-microsoft-hololens","tag-recent-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/124608","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=124608"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/124608\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/124609"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=124608"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=124608"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=124608"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}