Nvidia Develops Tools to Tackle Rogue AI Risks

Pushing back against the P(doom) crowd, Nvidia CEO Jensen Huang has argued that leading labs don’t need more regulation, just a little restraint and better tools. His company claims it can now provide the latter.

On Monday, Nvidia launched the Open Agent Safety Platform, a sandbox for training and observing new models and agents designed to prevent the kind of prison breaks that led to hacks at Hugging Face and the US government this summer. It’s a big new product for the chipmaker and the latest example of Nvidia expanding its product line beyond chips. 

Thinking Inside the Sandbox

“The first problem is the isolation; the containment wasn’t good enough. If the isolation and containment was good enough, that technology would be sitting in a lab, doing whatever it’s doing,” Huang said on an episode of The New York Times podcast The Ezra Klein Show when asked about the rogue AI models. The company’s new platform, he said on CNBC Monday, would create a “browser for agents” without any hidden escape hatch into the open web. He called misbehaving AI a solvable engineering problem, but allowed that if somehow it can’t be solved, we’re in big trouble. 

Add it to the list of Nvidia’s software and service offerings. The company agreed to acquire the aforementioned Hugging Face for $13 billion earlier this month in a bid to become the hub of open-source AI models and applications; the acquisition expands the company’s existing AI software library. The training sandbox, meanwhile, features two key tools that will continue to expand its chips-and-software flywheel:

• The first feature is OpenShell, which Nvidia had previewed in March and which runs on its Vera CPU chips. The open-source tool allows users to establish guardrails for what agents can and cannot access.
• The second feature is another open-source tool called Sentry. It runs on the company’s BlueField data processing units and can monitor agents in real time, quarantining those “that attempt to move outside their boundaries in milliseconds,” Nvidia said.

Buy, Buy, Buyback: Now that the leading frontier labs have established a market foothold, they must “shift their R&D … from just capability to a lot of verification, evaluation and testing,” Huang said before the product launch, predicting that “the amount of compute necessary to develop these models has increased by a factor of 10, because the evaluation is so rigorous.” In other words: more chip sales for Nvidia. In the meantime, the company also announced Monday that it plans to spend an additional $150 billion on buybacks, bringing its total remaining authorization to $235 billion. That would be the largest repurchase plan in corporate history, fitting for the largest company in history by market cap.