Sony Patent Highlights how Game Industry Should (and Shouldn’t) Use AI

Sony’s putting AI to work in the game studio. 

The tech giant filed a patent application for “generating a musical score for a game” on the fly using machine learning. Sony’s system aims to create and match music to real-time gameplay. 

Though some video games use “algorithmic music generation” to select snippets of audio from pre-generated sound samples, “the extent to which the soundtrack can be personalised is limited by the number of sound samples provided,” Sony said in the filing. 

Its tech aims to provide a soundtrack with a “greater degree of personalisation” using AI. Sony’s system would embed a “music generation apparatus” within a console, consisting of two parts: circuitry that generates the music and circuitry that picks up signals from gameplay. 

These signals would represent in-game actions, conditions, the current state of the game, and the movement from the pressing of buttons or joysticks. It could also pick up on physiological signals, movements, expression, or eye movement if a user is wearing an artificial reality headset. 

When the system receives a request to generate a new score, it feeds these signals to a machine learning model via “keywords” that represent the game’s state. Sony’s tech monitors gameplay to feed the model a stream of these keywords and update the score as necessary. 

This isn’t the first time we’ve seen Sony bring AI into the creative process. Sony Pictures CEO Tony Vinciquerra told investors at an event in late May that the company would use AI to cut film costs.

It’s also not the first time we’ve seen game developers take an interest in AI: Electronic Arts has sought to patent a way to playtest games using reinforcement learning; CEO Andrew Wilson claimed earlier this year that 60% of its “development processes” could be impacted by generative AI. A report from Wired in late July found that Activision Blizzard is using AI to develop some of its titles. 

But the use of AI in game development isn’t going over easy in Hollywood. Video game performers authorized a strike under SAG-AFTRA in late July, fighting for protections against the use of AI to replicate their work. They view the tech as a direct threat to their livelihoods.

Though AI can help speed up and enhance ideation, development, and testing in the industry, it shouldn’t be used as a substitute for the workers creating these games, said Yuqian Sun, the independent game developer of “1001 Nights” and an AI researcher and Ph.D. student at the Royal College of Art. 

This rings especially true when considering that lots of AI image-generators clash with copyright, an issue that AI firms like MidJourney and Stability AI have been battling in court in recent months. 

“There are definitely innovative and novelty chances for AI to bring new content and new experiences to the public,” she said. “But meanwhile, on the other side, the same technology can lead to the infringement of copyrights or authorship of creative labor.”