Google’s May Want to Keep Closer Track of User Hand Movements

Google may want to keep closer track of where you put your hands. 

The tech giant filed a patent application for gesture detection via “image capture of subdermal tissue” using a “wrist-pointing camera system.” Google’s tech determines hand movement using wrist-mounted cameras, working in tandem with a headset or pair of smart glasses for control of AR environments. 

The conventional method of tracking gestures uses camera sensors on head-mounted devices, which tend to eat up a lot of power, Google noted. “Due to this large power consumption … (using a) camera to deduce user gestures is that the world-facing RGB camera can only be used sparingly,” the company noted. 

Google’s solution to this is a wrist-worn near-infrared camera to calculate what the filing calls a “biological fluid flow metric” through the skin. This sensor would emit electromagnetic radiation to help it capture a sequence of 2D images to track changes in “perfusion,” or blood flow, between image frames that may indicate movement.

This data is fed to a trained neural network which determines the biological flow metrics of the images. Finally, the device’s gesture detection circuitry connects those metrics to specific gestures. 

To illustrate how this may look, a user wearing this device could point, pinch or make a fist and the system would pick up on it, reflecting those actions into interactions with an AR headset. Google notes that its system would be able to detect fine and minute hand gestures, such as telling the difference between an index finger-thumb pinching and a middle finger-thumb pinching.

Google’s innovations are in line with similar inventions from other tech firms, with the filing of Meta’s patent application for the “neuromuscular-signal-based detection” of in-air hand gestures, and Apple’s tech for interacting with virtual objects “using hand gestures.” Google’s filing adds more evidence to the idea that mixed-reality tracking is getting closer to users’ skin.  

And Google’s own patent activity is littered with artificial reality innovations, from monitoring brain activity to control extended reality systems to steerable cameras for head-mounted devices to tech replicates how light hits a pupil to make AR glasses less bulky. 

But Google may be struggling in the wearable tech department: The company reportedly cut hundreds of roles late last week, slashing a number of employees on its AR team, as well as reorganizing its FitBit, Nest, Pixel and Assistant divisions, the company confirmed to several  media outlets.

Despite the cuts, a company spokesperson told 9to5Google that it “continues to be deeply committed to other AR initiatives, such as AR experiences in our products, and product partnerships.”

Depending on what these divisions decide to prioritize, the future of the tech in this patent could be up in the air.