Microsoft’s smart backpack, Disney’s AI roleplaying & Facebook’s music recommendations

---

1. Microsoft – AI wearable backpacks

Microsoft is looking into AI-assisted smart wearables that work as a hands-free digital assistant. Specifically, the patent filing explores a smart backpack.

Why backpacks?

Microsoft specifically wants to build digital assistants that people can carry with them as they go about their life, without requiring theem to divert their attention away from whatever task they’re performing. For instance, with digital assistants on a smartphone, users need to stop what they’re doing, obtain their device, interact with the digital assistant (either with voice or touch), and then put the device away. Microsoft sees this as a limiting factor to the usefulness of assistants.

Microsoft also wants the assistant to be ‘context aware’. Through cameras and microphones on thee backpack, the assistant can better understand the user’s surroundings, and therefore any user commands can be understood in their proper context.

For example, imagine a user is on the ski slopes with the smart backpack. If they’re not sure which direction they’re allowed to travel, they could ask the backpack. Once this question is picked up and processed by the microphone, the backpack’s camera could be activated in order to understand the environment. The AI assistant could then work out where the user is, access a ski resort database, determine which direction is allowed, and then direct the user to where they are allowed to ski.

Besides helping out people skiing, Microsoft also imagine the wearable benefitting people with disabilities, particularly people who are visually impaired. For instance, the backpack could help a blind person navigate a busy city, providing haptic or auditory feedback for where to turn, or when it’s safe to cross the street.

In the filing, Microsoft set out to address any privacy issues that arise from users carrying cameras and microphones with them in a backpack – all video recordings will be stored temporarily and user-specific data will be encrypted. To be honest, this is more than I’ve seen on other filings from the likes of Facebook and Amazon.

I’m curious to see if this is actually a product that Microsoft is looking to release into the world, or if it is instead an internal project being explored. Only time will tell.

2. Disney – AI role playing experiences based on selected scenarios

Disney patent applications are always fun to look at. Theyre a product of a company looking into the future through the the lenses of entertainment and human connection.

In this latest application, Disney outlines a AI chat bot system that can play as an intelligent character in a role-playing game with kids.

For example, in one outlined scenario, users would create a role-playing scenario, where they choose a role to play (e.g. wizard, warrior or ninja), and an objective (e.g. infiltrate a secure area, interrogating a character for information, or flattering someone).

As the user interacts in the scenario, the AI system would use natural language processing to identify keywords, analyse sentiment, evaluate the intent of a user, and evaluate how ‘close’ the user is to the role that they are supposed to be playing.

Where this system will live isn’t really outlined in the filing. But if we parse through some of the old Disney patents we’ve explored in previous editions, we’ve seen that Disney is working on robot actors that can interact with visitors at a theme park and smart toys that encourage kids to be fit. This role-playing AI system could in theory apply in both of these products.

Disney is definitely an underrated vehicle for bringing cutting edge technology to the masses and humanising it.

3. Facebook – music recommendations for videos

No fancy graphic for this one unfortunately.

This Facebook filing looks at generating music recommendations for users to add to a piece of video content.

When a user uploads a video, Facebook will use machine learning to extract specific features in a video – basically identifying what concepts, objects and visual characteristics are in the video.

For example, a video could be an event such as a birthday or a wedding. The video could be taken during a specific time of day, such as the morning or night. The video might feature some scenery, such as a beach or a sunset. A location could be identified such as a beach or a city. An object could be identified such as a face or a vehicle.

On the music side, Facebook will extract features for each song, including the song title, artist, genre, lyrics and the spectograms (i.e. the sound wave of a song).

For each uploaded video, Facebook will basically map the video feature embeddings to a set of music feature embeddings, and identify songs that are closest in similarity to what the video is about.

Why is this interesting?

In a previous issue, we looked at a Facebook patent filing outlining the creation of music charts to facilitate more music discovery, probably for Instagram Reels and Stories.

In this filing, it looks like Facebook might be wanting to reduce the effort required for people to create engaging videos. In theory, this helps more people create videos and if the right song can elevate how engaging a video is, it also makes scrolling through the Reels feed / Stories more engaging.

Whether it’s purely for competing against TikTok or Facebook is looking to compete against Spotify in the long-run, these filings are a clue that Facebook is exploring how to deeper integrate music into their ecosystem.