How AI-Powered Robotics Face Obstacles in Real-World Scenarios

Tech firms want to bring AI into the real world, but they may not be ready for each other just yet. 

Nvidia this week announced a host of new models, libraries and infrastructure for robotics developers. One model, called Cosmos Reason, is a 7-billion parameter vision language model offering “world reasoning” for synthetic data curation, robotic decision-making and real-time video analytics. 

“By combining AI reasoning with scalable, physically accurate simulation, we’re enabling developers to build tomorrow’s robots and autonomous vehicles that will transform trillions of dollars in industries,” Rev Lebaredian, vice president of Omniverse and simulation technologies at Nvidia, said in the announcement. 

This isn’t the first time Nvidia and other firms have sought to bring powerful AI models into robotics contexts. In June, Google DeepMind introduced Gemini Robotics On-Device, a “vision language action” model that it claims can help a robot adapt to new domains in between 50 and 100 demonstrations. Plus, both Google and Nvidia have filed plenty of patent applications for AI-powered robotics tools, including data simulation, control and safety measures. 

Still, deploying AI – particularly powerful, foundational language models – in robotics contexts comes with a number of obstacles in materials science, mechanical engineering and software, said Ariel Seidman, CEO of edge AI firm Bee Maps.

• While it’s possible to teach a robot to do a very specific task, creating a general-purpose robot, controllable by a large language model and capable of handling a wide array of tasks, is a “much more open-ended problem,” said Seidman.
• “There’s almost like an endless number of scenarios that it has to deal with,” said Seidman. That unpredictability can be difficult to account for in training. 

“It’s just a totally uncontrolled environment where I don’t think people are going to have the patience for robots that are constantly misbehaving or not meeting expectations,” said Seidman. 

Logistics, factories and warehouse environments may be a good place to start for a number of reasons. For one, the environments themselves are often tightly-regulated, so the robots operating in them will face less unpredictability. These environments are also a great way to build up a “massive corpus of data” at scale with fewer privacy concerns than in public or personal use cases, said Seidman.  

“There’s all the etiquette, combined with the utility that needs to be embedded within a robot,” said Seidman. “You need that kind of data so that robots can understand how to respond to a human in these situations.”