Google Patent Could Help Robots Have Better Memories

Google wants to help robots strengthen their memories. 

The company aims to patent a system that controls robotic agents “using scene memory data.” Google’s tech essentially helps robots have better decision-making skills by using neural networks to comprehend both historical data and the scenes in front of them. 

“Many complex tasks … require the agent to perform tasks over a long time horizon,” Google said. “Hence, being able to properly capture past observations is crucial to achieving good performance on the task.” 

To start, Google’s system would collect data about an environment through cameras or LIDAR sensors and convert them into embeddings, or structured data that a neural network can actually use. This information is added to a larger pot of “scene memory data,” or historical data with details on the environment’s dynamics. 

Next, an encoder neural network with a “self-attention mechanism” takes in the scene memory and picks out relevant information from its past observations, creating an “encoded representation of the scene memory data.” Finally, that encoded representation is sent to a “decoder” neural network, which essentially creates an action plan for the robotic agent. 

Google claims its tech allows these robots to “effectively capture long term dependencies and learn a relevant geometry of the environment,” as well as “prioritize particular observations at any given time step without requiring any pre-determined structure of the environment to be known in advance.” 

Plenty of big tech firms have been hard at work researching ways to fuse AI and robotics to make these machines safer, more observant, and more reactive. Google DeepMind has also recently tested bringing Gemini’s capabilities to its robots, letting them handle natural language instructions and potentially fulfill tasks outside of simple navigation.

This patent could offer a novel way of helping autonomous machines learn and improve: long-term memory, said Rhonda Dibachi, CEO of manufacturing-as-a-service platform HeyScottie. “In order to be truly intelligent, you have to have both short-term — i.e., working — memory as well as long-term — i.e., situational — memory, and this is a way to do that,” Dibachi said. 

Long-term memory allows autonomous machines the benefit of “situational intelligence,” said Dibachi. It’s the difference between a robot going into a space aware of the objects, dimensions, and people around it, and going in blind. “Rather than having to feed a robot all the information for the starting conditions … you can have a robot that remembers from yesterday.” 

There is a lot of money to be made in making smarter robots, said Dibachi. Though these machines currently aren’t capable of hands-off operation, the potential productivity that autonomous logistics and production may hold is an attractive proposition to manufacturers and tech firms alike. 

Though Google isn’t new to the world of robotics and autonomous machines, hardware is far from its main focus — it continues to plunge resources into software and large language model development. But tech like this could provide an avenue to access a valuable, vast well of real-world data, said Dibachi, whether it be from self-driving vehicles, factory robots, or humanoid co-workers.