Big Tech may be building even bigger AI, but it’s unclear how useful these massive language models are — or if, in fact, bigger is better at all. 

More and more, enterprises are choosing small language models over their foundational, hundred billion-parameter counterparts. These models, which can be as small as a few million parameters, are often a faster, cheaper and more efficient alternative for enterprises that are still figuring out their AI strategies. 

“Not every enterprise will have the infrastructure to operate [large language models] for every single need,” said Andrew Joiner, CEO of AI automation platform Hyperscience. “In a lot of ways, it’s like taking a helicopter to fly across the street.” 

Most enterprises are using generative AI in some form or fashion. According to a survey of more than 500 IT leaders from Hyperscience and The Harris Poll, 79% of organizations reported using generative AI, and 97% said they plan to integrate it further in the next two to three years. But as it stands, these enterprises are still trying to grasp how generative AI fits into their operations – and many are still wary of the technology’s pitfalls. 

• Data privacy remains a major obstacle to AI adoption: Around 47% of IT leaders reported privacy concerns as the primary barrier. Because of this, many are underutilizing the data they have available, with 3 in 5 organizations not leveraging their entire data estate. 
• Integrating these models into core tasks in a meaningful way would mean giving them access to a company’s “most sensitive” data, such as legal contracts, purchase orders, user data and more, said Joiner. “They’re not appropriate to put into and use with a large language model.” 
• Cost also remains an issue. Around 36% reported the high startup expenses made them wary of increasing AI integration, and 34% said high ongoing costs were a concern. 

Thinking small may help enterprises solve these problems. With a small, locally run model, businesses have better traceability and data security control over what goes in and comes out of it, said Bob Rogers, Ph.D., the co-founder of BeeKeeperAI and CEO of Oii.ai. “You know exactly where your data is going, and that’s comforting,” he said. “For a lot of very sensitive data, local models are the right choice.” 

Plus, with only tens of millions parameters to train as opposed to hundreds of billions, these models are far easier to put together quickly, are good for specific tasks, are more power efficient and can be more cost effective, said Joiner. That allows enterprises to “maximize their outcome” in building AI in the first place. 

Enterprises seem to be aware of this benefit: Of the IT professionals who reported using generative AI in the survey from Hyperscience and The Harris Poll, three in four organizations said small language models outperformed large ones in areas such as speed, cost and return on investment. Tech giants, too, seem to see the benefits of going small, with Microsoft, Amazon, Meta, and Google all researching or releasing their own lightweight models in the past year. 

But smaller models aren’t a panacea for IT’s AI woes, Rogers said. Simply put, these models are “not always as smart as the bigger models,” he added, and they tend to hallucinate more. They also tend to lack the people skills that large language models cultivate from billions upon billions of data points, Rogers noted, often making them less friendly to interact with. “They sometimes don’t have quite the level of nuance in their understanding of how ideas flow.” 

That lack of nuance means that enterprise professionals have to pick and choose when small or large language models might be a better fit. While smaller models may not be the best choice for automating HR conversations or customer service interactions, they could be well suited for creating personalized AI assistants or helping with specialized research and development. 

“You need to pick the right tool to get to the outcomes you need,” said Joiner.

When a chatbot can generate code in the blink of an eye, where do developers fit in? 

The AI revolution has brought with it new ways to automate and augment just about every task, a major one being code-writing. Tons of tech giants have introduced a code-generation tool in the past year or so, including Microsoft, Amazon, IBM, and Google. 

An examination of recent patent applications also reveals an interest in making AI code generation smoother. Google, for example, is seeking to patent a system for “iterative neural code translation,” which improves the quality of automatic translation between code languages by identifying bugs and iteratively fixing them over several tries. This tech uses neural networks to make sure things don’t get lost in translation – literally. 

It also falls in line with a number of other code generation patents that Google has sought in the past, as well as Google’s own internal push to create more code with AI: CEO Sundar Pichai said in the company’s third-quarter earnings call that more than a quarter of new code for its products is initially generated by AI before being reviewed by engineers. “This helps our engineers do more and move faster,” Pichai said. 

This strategy begs the question: When is the time and place to use an AI code generator? Though these tools are generally reliable, they still have the tendency to occasionally hallucinate and make mistakes the same way every generative AI product does, said Kaj van de Loo, chief product and technology officer and chief innovation officer at UserTesting. 

For enterprises, it’s important to think about what that code may be used for, he said. For instance, if you need to write code that’s “super mission-critical and needs to perform flawlessly,” it may not be the best use of code-generation software. But for everyday code-writing, he said, “There are some areas that are so right for this, where I think anybody not doing it is missing out big time.” 

But in any case, it’s vital that a human remains in the loop, said van de Loo. Though we may soon get to a place where humans and machines can reliably work together to check code, these models can’t replace developers: Researchers in China recently discovered that ChatGPT is overconfident in the quality of its code when checking its own work. 

“In the AI tooling space, there are two approaches to delivering value: One is to augment humans, and one is to replace humans,” said van de Loo. “With our code-generation, we’re still pretty firmly in the augmentation camp. A human needs to take responsibility and own the code.”