Snap’s Patent Could Help Advertisers Know What Makes Users Swipe

Snap wants to understand its audience better. 

The social media firm wants to patent a system for “multimodal sentiment classification.” Snap’s invention essentially creates AI that can glean emotional context from both text and image data. 

Snap said that conventional methods of sentiment detection, which involve just textual data, “tend to be less effective when applied to some types of network site posts, such as a social media post comprising an image and a short caption, as such posts have limited text to analyze.” 

Snap’s system aims to cover all of the data within “multimodal messages,” including the image itself, the text in the caption and symbols such as emojis within captions. The system uses multiple neural networks equipped with what’s called “long short term memory,” which make them able to selectively remember or forget different information over time. 

Two neural networks are used to analyze the text itself to understand it in context of the image behind it. The image itself is analyzed using an AI-based “visual attention mechanism,” which extracts “valuable visual features” that are closely related to the text. 

The data from all three models is then used to generate the likelihood of a positive, neutral, or negative sentiment. So, if a user posts a photo of a movie poster with the caption “so excited to see this movie” onto their Snapchat story, this system would determine that the poster has a positive sentiment related to that particular movie. 

This data could be useful for enhancing the user experience, said Mithru Vigneshwara, product director at QReal.io. This could look like personalizing emojis and reactions to content, as well as pushing certain content where users have positive sentiments. “Let’s say the system realizes that something has a positive sentiment versus a negative sentiment, your default reactions could change automatically,” he said. 

But figuring out the positive and negative associations a user has with certain topics, people or places could be helpful for the company’s biggest moneymaker: advertising. 

Snap has struggled in an unpredictable ad market over the past year. It had posted two quarters straight of declining revenue when it finally saw an uptick of around 5% in its third-quarter earnings report. CEO Evan Spiegel noted in a letter to investors that the company is working on boosting its ad platform to offer a better return on investment for its advertisers. And according to The Verge, one of Spiegel’s stretch goals for this year is 20% year-over-year ad revenue growth. 

Snap may be able to better target ads with a deeper understanding of what users like and don’t like, and what positive sentiments extend from that, said Jake Maymar, AI innovation lead at a stealth startup. “You could start seeing these webs of advertisers that are natively advertising on this platform, and you may not even realize it, but you’re going to start having really positive associations,” he said. 

But this doesn’t consider the different ways people emote on the internet – particularly, sarcastic teens and young adults. For example, how would a system like Snap’s account for a caption that says “I love my healthy dinner” when the image is a photo of a Lexapro tablet, some gummy bears and a Diet Coke? 

One solution could be taking an entire user profile into account, not just the posts themselves, said Maymar. If a user has a history of posting satirical content, then the system may understand that their posts are meant to be humorous or sarcastic. 

But with the speed at which internet vernacular changes, this system may need to be closely plugged into current events and pop culture to keep up, said Vigneshwara. “One day a word might be something positive, and in a month the whole thing could be negative,” he said. “It could be interesting to see how exactly this is going to react to that.”