Apple Puts Spatial Computing in Users’ Hands

Apple may be paying close attention to the way you move your hands. 

The company filed a patent application for a system to interact with virtual objects “using hand gestures.” Essentially, Apple’s patent details a system that can pick up on hand gestures and controls in a “more efficient and intuitive” manner. It does so through sensors — many of which are already part of available Apple devices — ranging from motion-capture sensors to temperature sensors to “peripheral device” sensors (i.e., home appliance or smart speakers).

The 90-page filing details several ways that a user could control various Apple products with hand gestures, including with watches, TVs, laptops, tablets and head-mounted displays. The filing even mentions that these controls could be implemented in windshields and windows with “integrated display capability.”

One example that Apple’s filing details are different ways in which a smartwatch can integrate with several types of displays, such as guiding controls on a smart TV display, or controlling virtual objects on a tablet. The definition of a “virtual object” is also quite wide, noting that elements such as “digital images, video, text, icons, and control elements such as buttons and other graphics” fall under this umbrella. 

Apple noted that conventional systems are often cumbersome and require complex inputs to achieve certain outcomes in augmented reality environments. These systems are often tedious and error-prone, Apple said, creating “a significant cognitive burden on a user, and detract(ing) from the experience.” 

“In addition, these methods take longer than necessary, thereby wasting energy of the computer system,” Apple said. “This latter consideration is particularly important in battery-operated devices.” 

As Apple noted in the filing, there are several use cases for advanced controls such as these, even hinting at integrations for the rumored Apple Car. But given the patent’s focus on the Apple Watch’s potential, the device could be a key component to the company’s wider connectivity plans. 

Apple already offers some gesture recognition through the Apple Watch as part of an accessibility setting called Quick Actions, which allows users to respond to alerts with movements such as gripping a fist or making a pinching motion with your fingers. This patent may expand on already-existing capabilities. 

That said, likely the biggest piece of Apple news so far this year was the debut of its Vision Pro headset. The long-awaited device already boasts eye-tracking as a means of controlling your AR environment. Enabling hand tracking through a connection with the Apple Watch could be an added bonus. 

Apple already has a deeply connected ecosystem of devices, but the ease of use and connectivity of this patent could lock in customers to the Apple brand even more than its devices already do. If all of a user’s devices are Apple, and all of those devices are connected with one another and can easily interact with one another, a user has less incentive to purchase, say, an Android phone, an Intel PC or a Meta headset.  

“In the near term, I think Apple wants users to use all their devices together,” said Jake Maymar, VP of Innovation at The Glimpse Group. “The competitive advantage that Apple has over a lot of other companies is that all the devices will work seamlessly together.”

But this patent, added to Apple’s current line of devices and patents, could point to a larger goal from Apple, said Maymar: a fully connected ecosystem of devices controlled through spatial computing, or a way of interacting with computing in the space around you that was introduced in the Vision Pro. And because Apple has a large, core user base of dedicated customers, he noted, the company may have an easier time getting users to adopt this technology. 

Have any comments, tips or suggestions? Drop us a line! Email at admin@patentdrop.xyz or shoot us a DM on Twitter @patentdrop. If you want to get Patent Drop in your inbox, click here to subscribe.