+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Computers (https://www.sickgaming.net/forum-86.html)
+--- Forum: Apples Mac and OS X (https://www.sickgaming.net/forum-87.html)
+--- Thread: AppleInsider - Apple AR research encompasses so much more than entertainment (/thread-97299.html)
AppleInsider - Apple AR research encompasses so much more than entertainment - xSicKxBot - 09-17-2020
Apple AR research encompasses so much more than entertainment
<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/09/apple-ar-research-encompasses-so-much-more-than-entertainment.jpg" width="800" height="547" title="" alt="" /></div><div><p>A series of patent applications show that Apple is focused on how users can work in an augmented reality or virtual environment, with the company working on practical sides of how to make that space feel more real.
</p>
<div class="col-sm-12">
<p>Maybe we’ll get <a href="https://appleinsider.com/articles/20/09/14/apples-wearable-ar-hardware-ambitions-may-not-start-with-apple-glass">goggles first</a>, or maybe “<a href="https://appleinsider.com/inside/Apple-Glass">Apple Glass</a>” will be as slim and stylish as we hope. Ultimately, though, Apple is seeing <a href="https://appleinsider.com/inside/Apple-AR">Apple AR</a> and VR as being used for more than the games we see in goggles, or the notifications we might see in glasses. </p>
</div>
<div class="col-sm-12">
<p>Instead, it’s focusing on how users could get work done within an environment that is partly or wholly computer generated. That includes how we might use applications that only exist in our AR world, it includes how sound can help make an all-encompassing experience. And it also looks at how we look around, how we can find our way when half our view is virtual and half is real.
</p>
</div>
<p><h2>Audio in AR</h2>
</p>
<div class="col-sm-12">
<p>One of the newly revealed patent applications, “3D audio rendering using volumetric audio rendering and scripted audio level-of-detail,” wants to help users experience a more <a href="http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=7&p=1&f=G&l=50&d=PG01&S1=(apple.AANM.+AND+20200917.PD.)&OS=aanm/apple+and+pd/9/17/2020&RS=(AANM/apple+AND+PD/20200917)">natural-seeming environment</a>. It does this through how we may hear the sounds of the AR world, and Apple is also concerned with how programmers can make better, easier use of audio.
</p>
</div>
<div class="col-sm-12">
<p>“Computer programmers use 2D and 3D graphics rendering and animation infrastructure as a convenient means for rapid software application development…, ” begins Apple. “One challenge for such graphical frameworks is that graphical programs such as games often require audio features that must be determined in real time based on non-deterministic or random actions of various objects in a scene.”
</p>
</div>
<div class="col-sm-12">
<div class="align-center"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/09/apple-ar-research-encompasses-so-much-more-than-entertainment.jpg" alt="A river can't be rendered as a single point of sound" class="img-responsive article-image" loading="lazy">
</div>
<p><span class="carousel-caption">A river can’t be rendered as a single point of sound</span></p>
</div>
<div class="col-sm-12">
<p>So a game might have an explosion, but if you’re looking away from that, the sound of it has to appear behind you. “Incorporating audio features in the graphical framework often requires significant time and resources to determine how the audio features should change when the objects in a scene change,” says the patent application.
</p>
</div>
<div class="col-sm-12">
<p>Apple says that currently sounds are tied to specific points in the environment, such as that explosion being placed where the correct graphics are. But while this is typical of AR, it’s not always ideal and it is always complicated.
</p>
</div>
<div class="col-sm-12">
<p>“[It] usually means that an application is required to generate points for each of various sounds that exist in the virtual audio environment,” says Apple. “This process is complex, and current approaches are typically ad-hoc.”
</p>
</div>
<div class="col-sm-12">
<p>The drawings in the patent application show, as an example, a user being faced by an AR river. There is no one single point of origin for the sounds of the water, it is a continuous area, a region within the AR environment. </p>
</div>
<div class="col-sm-12">
<p>Apple’s solution is to “represent a sound source in a 3D virtual audio environment as a geometric volume, rather than as a point.” The patent application discusses many methods and issues to do with replacing a point with an overall region, but the purpose is always to create this more natural sound environment.
</p>
</div>
<div class="col-sm-12">
<p>It could also be something that Apple AR itself, as the kind of AR operating system, might provide instead of every developer having to create it.
</p>
</div>
<div class="col-sm-12">
<p>“This makes it possible for an audio rendering engine, rather than an application program, to use the geometric volume of the object to render a more realistic audio environment,” says Apple. </p>
</div>
<p><h2>Directing your attention</h2>
</p>
<div class="col-sm-12">
<p>If there were an explosion in a game, you are most likely to look in its direction but it is entirely up to you. Apart from when there are specific movements, or sounds, though, AR can present a rather unvarying view.
</p>
</div>
<div class="col-sm-12">
<p>As you look through any AR system, Apple says that what you can see will range “from wholly synthetic experiences to barely perceptible computer-generated media content superimposed on real-world stimuli.” You might have an entire world in VR, or you could have the smallest of “turn here” signs as you walk down a real street.
</p>
</div>
<div class="col-sm-12">
<div class="align-center"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/09/apple-ar-research-encompasses-so-much-more-than-entertainment-1.jpg" alt="Apple's system could allow an AR application to put a glow around where you're supposed to look" class="img-responsive article-image" loading="lazy">
</div>
<p><span class="carousel-caption">Apple’s system could allow an AR application to put a glow around where you’re supposed to look</span></p>
</div>
<div class="col-sm-12">
<p>An issue that Apple wants to address with <a href="http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=29&p=1&f=G&l=50&d=PG01&S1=(apple.AANM.+AND+20200917.PD.)&OS=aanm/apple+and+pd/9/17/2020&RS=(AANM/apple+AND+PD/20200917)">another patent application</a>, is how the user knows where to look. When they are surrounded by VR, or when there is a single AR object by them, there needs to be a way to direct the user’s attention when needed.
</p>
</div>
<div class="col-sm-12">
<p>“Attention direction on optical passthrough displays,” concentrates on AR where the computer-generated reality (CGR) of what you’re seeing is a combination of the real world and this superimposed media content. </p>
</div>
<div class="col-sm-12">
<p>“[It] may be desirable to direct the attention of a user in a CGR environment to an object or area of the CGR environment,” says Apple. “For example, a game executed by an HMD [head-mounted display] presenting the CGR environment may desire to direct the user’s attention to a game objective.”
</p>
</div>
<div class="col-sm-12">
<p>“As another example, a map application executed by the HMD may desire to direct the user’s attention to a map destination,” it continues.
</p>
</div>
<div class="col-sm-12">
<p>Interestingly, Apple is also looking at when it would be best to prevent a user’s attention being drawn to something. “[It] may be desirable to direct the attention of a user away from an object or area of the CGR environment, such as an advertisement or inappropriate context.”
</p>
</div>
<div class="col-sm-12">
<p>Whatever the motivation for attracting attention or avoiding it, Apple’s proposal comes down very broadly to two options. It can display “noise in areas of the CGR environment where attention of the user is not to be directed,” for instance.
</p>
</div>
<div class="col-sm-12">
<p>Or it can use what Apple describes as masking. If an application needs a user to look at an AR pot plant on a table, that plant is a graphical image superimposed onto reality. An attention system might add a masking element “behind” the object, and then add a glow to it, for instance.
</p>
</div>
<p><h2>Working tools in AR</h2>
</p>
<div class="col-sm-12">
<p>One item that is very likely to be present in an AR environment, and to require a user’s attention, is an application, a tool. AR can turn any surface into a calculator, for instance, or a screen.
</p>
</div>
<div class="col-sm-12">
<p>A third patent application, newly uncovered, is simply called “Displaying Applications,” and it involves making an AR tool <a href="http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=22&p=1&f=G&l=50&d=PG01&S1=(apple.AANM.+AND+20200917.PD.)&OS=aanm/apple+and+pd/9/17/2020&RS=(AANM/apple+AND+PD/20200917)">fit into the environment</a> around the user. “The techniques include determining a position of a physical object on a physical surface,” says Apple, “[and] displaying a representation of an application in a simulated reality setting.”
</p>
</div>
<div class="col-sm-12">
<div class="align-center"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/09/apple-ar-research-encompasses-so-much-more-than-entertainment-2.jpg" alt="How real objects could be made AR tools" class="img-responsive article-image" loading="lazy">
</div>
<p><span class="carousel-caption">How real objects could be made AR tools</span></p>
</div>
<div class="col-sm-12">
<p>It also involves “modifying attributes of the representation of the application in response to detecting changes in the position of the physical object on the physical surface,” such as when an object is picked up.
</p>
</div>
<div class="col-sm-12">
<p>This is similar to the recently revealed patent application regarding <a href="https://appleinsider.com/articles/20/07/21/apple-glass-could-turn-any-surface-into-a-display-with-touch-controls">turning surfaces</a> into AR controls. This new application is also credited to Samuel Lee Iglesias, whose related work includes <a href="https://appleinsider.com/articles/20/04/02/apple-researching-trackpad-gesture-combo-for-controlling-ar">using trackpads</a> to manipulate AR objects.
</p>
</div>
<p><h2>Apple’s plans for AR</h2>
</p>
<div class="col-sm-12">
<p>As noted by <em>AppleInsider</em> before, Apple’s AR work is clearly <a href="https://appleinsider.com/articles/20/07/23/apple-plans-to-dominate-all-aspects-of-ar-with-apple-glass-and-more">not confined</a> to one single product. Rather, the company seems to be focusing on it becoming a huge shift in how all of us use our devices.
</p>
</div>
<div class="col-sm-12">
<p>It’s possibly still the case that a head-mounted display is automatically associated with gaming, but Apple is putting considerable effort into making a technology we can all actually work with.
</p>
</div>
</div>
https://www.sickgaming.net/blog/2020/09/17/apple-ar-research-encompasses-so-much-more-than-entertainment/