Microsoft’s XR Control Patent May Get Too Close for Comfort

Microsoft is taking tongue-in-cheek a little too literally. 

The company filed a patent application for “inertial sensing of tongue gestures.” Microsoft’s filing lays out a system for detecting tongue movements as a means of controlling an artificial reality device. 

“Touch and hand gestures have certain limitations for applications where the user’s hands are occupied or for users with certain physical limitations,” Microsoft said in the filing. Speech recognition and gaze tracking “also have certain limitations,” the filing noted, such as privacy drawbacks and latency issues. 

Microsoft’s tech tracks motion signals using an “inertial sensor” to detect tongue gestures. Microsoft notes that these sensors would be “non-obtrusive” and externally worn, such as within a head-worn display, earbuds, headphones or cochlear implants. (This overcomes the invasive nature of other tongue-tracking inventions, the company said, such as mouth-worn retainers or “electromyography sensors.”) 

Once this system detects these motion signals, a machine learning model would take that data and translate it into gestures that are reflected in a display. In practice, this may allow a user to control elements of an artificial reality scenario without speaking or using their hands. 

For example, when typing, this system may use gaze-tracking to hover over letters on a virtual keyboard, and tongue gestures to select them. In an audio-only application, these gestures may allow a user to switch songs, pause music, or interact with a virtual assistant. 

This offers the user “quiet, hands-free input” to their devices that can be performed with “little or no perceptible external movement by the user and can be performed by users with serious physical limitations.” 

As outlandish as it seems that tech firms want to get into your mouth, this isn’t the first time Microsoft has taken an interest in getting close to its users: The company previously sought to patent “high-sensitivity” facial tracking for artificial reality devices.

Microsoft also isn’t the only company that’s looking for hands-free ways to control artificial reality. Meta has looked at ways to track your body’s neural and muscular signals, Apple has filed patents for eye tracking using neural networks, and Snap’s filed a patent for methods to track gaze direction. 

Microsoft offers a line of mixed reality headsets called the HoloLens that start at a price tag of around $3,500 and are mainly sold as an enterprise offering, but metaverse tech is far from its main focus. Plus, the company has competition in the space with Meta’s line-up of more affordable Quest headsets, and Apple’s recent drop of the Vision Pro. As mixed reality tech continues to forge onward, Microsoft could make cash by licensing this tech to other operators.

The goal of these methods of tracking is to create a more intuitive user experience, aiming to scale adoption and increase the amount of time that these devices are strapped to people’s faces. Plus, a positive to these hands-free experiences is expanding the demographic of people that can use them, creating access for people with certain physical limitations or disabilities that may have constraints to their range of movement. 

But a common thread with these technologies is that they walk the delicate line of weighing privacy versus convenience. Equipping these headsets with the capability to constantly read body signals may simply make users uncomfortable, though it may venture to do the opposite. And if the data processing for these experiences isn’t done locally, rather via a remote server, that creates another avenue for privacy problems.