The list of companies creating technologies that could reduce the industry’s reliance on Nvidia might be longer than a shopping list for making a traditional mole poblano.

Among Big Tech firms, Meta rolled out four new generative AI chips in March that it said will lead to cost savings while still competing on a tech level with rivals’ GPUs. Two months earlier, Microsoft launched its Maia 200 chip, which is focused on inference (tasks such as answering queries and creating Studio Ghibli-style selfies). SpaceX, meanwhile, plans to invest between $55 billion and $119 billion in designing and manufacturing AI chips with Intel’s help.

And that’s not counting Google and Amazon; OpenAI and Anthropic, the companies that created AI’s best-known chatbots; Cerebras, which raised $5.5 billion in the year’s biggest initial public offering so far, and startups trying to break into the market.

They have a viable entry point: Now that AI has its training wheels off, the types of chips that will keep its momentum rolling are different from the ones that gave it the first push forward.

Majestic Labs Co-Founder and President Sha Rabii told The Daily Upside that AI is at a tipping point, with more work focused on inference. GPUs like Nvidia’s specialize in training AI, and companies are looking for new options that can more efficiently run AI after the models have been trained.

For Majestic Labs, the best way to make AI more efficient is to find a cost-effective way to increase memory capacity. Memory has become an expensive pain point for the AI sector, which is facing a shortage of high-bandwidth memory (HBM) needed to run the models it spent billions to create.

Making Memory

GPUs from Nvidia have incredible compute power, but relatively little memory, Rabii explained. That creates a bottleneck: “All that compute is just sitting there idle because you’re not able to feed the computational engines with the data they need to be running,” he said. “The compute isn’t doing anything because it’s waiting for data to come from memory.”

Majestic believes most new inference chips don’t go far enough to solve the memory problem. Its Prometheus server system, leveraging its own chip, supplies 1,000 times the memory capacity of GPUs like Nvidia’s, Rabbi said. The system relies on less expensive Dynamic Random Access Memory (DRAM) rather than HBM.

While companies have tinkered with creating 3D chips layered like silicon lasagnas to pack more memory in and solve the supply problem, Rabii contends these chips can run too hot, since cooling has to penetrate multiple layers.

Keeping chips cool is another large line item on companies’ budgets. When people talk about AI using millions of gallons of water, they’re talking about cooling data centers, which is typically done by evaporating water into the air. The next generation of AI chips, including Nvidia’s, comes with creative cooling solutions to further cut costs.

Better Together

Nvidia’s GPUs won’t become obsolete overnight. Most of the chips being made by competitors aren’t replacements for them, and they aren’t trying to be. Instead, the new wave of chips often supplements Nvidia’s offerings, so companies can buy fewer of them rather than nix their GPU budgets altogether.

Google and Amazon seem the closest to stepping on Nvidia’s toes as they rake in tens of billions in chip-related revenue:

  • Google has been making AI processors for more than a decade, and last month, it said it’ll create specialized chips that separate inference work from AI model training. Its TPUs (Tensor Processing Units) are used by Citadel Securities as well as Anthropic, which recently signed an expanded deal to tap multiple gigawatts worth of computing capacity from them. Broadcom, which helps Google make chips, said the products have garnered tens of billions of dollars in revenue.
  • Amazon, meanwhile, makes a handful of AI chips, one of which it markets as an affordable Nvidia alternative. Its Trainium chip has raked in billions, according to CEO Andy Jassy. Meta agreed last month to use tens of millions of Amazon’s Graviton CPUs.

In a related development, both Google and Amazon signaled in late April that they’re considering selling their chips directly to customers. Previously, they were only accessible through the companies’ respective cloud services.

OpenAI has joined the fray by making custom-designed chips in collaboration with Broadcom, and Anthropic was reported last month to be considering designing its own chips.

China Cuts In

Cerebras, which builds inference-focused chips that are used by both Amazon Web Services and OpenAI, jumped 68% Thursday in its first day of trading. Groq (no relation to Elon Musk’s Grok chatbot), meanwhile, attracted a $17 billion deal with Nvidia for its chip tech. Nvidia is prepping a version of Groq’s chip that could be sold in China, Reuters reported, even though Nvidia’s most advanced chips have been barred from the country due to defense concerns.

Chinese rivals like Huawei and Cambricon are trying to offer Nvidia alternatives, but Chris Miller, author of Chip War: The Fight for the World’s Most Critical Technology, told the Daily Upside they’re miles behind Nvidia in tech and production capabilities. Miller said domestic competition is significantly stronger, from both Big Tech giants and startups with strong inference architecture. Hence, Nvidia scooping up Groq.

Still, most of the tech giants in the large list of companies creating their own chips keep buying GPUs from Nvidia, and that might make AI better in the long run. To create the ultimate data-center tech stack, companies are combining multiple chip types with distinct superpowers (low latency, high throughput and so on). If companies tried to use just one system, Rabii said, “You end up compromising and not being great at any of it. You sort of just average out.”

So while Rabii does expect a shakeout at some point, he believes there will still be room for more than just Nvidia.