Microsoft Gets Virtual (plus more from Square & Disney)

Roaming in VR, Video analysis of food service counters, and Gaze tracking displays

1. Microsoft – Reality guided roaming in VR

Microsoft are thinking about how virtual environments in VR can dynamically update to reflect the physical environment of a user.

This is to ensure the ability of a user to safely roam their physical environment without colliding into real-world objects, while also remaining fully immersed in the virtual world.

In the illustrations above, the images on the left of each pairing (210A, 220A, 230A) reflect different physical environments that a user would see if they took off their headset.

In each, there are various obstructions – e.g. physical walls, chairs, tables etc.

Microsoft wants to use sensor data to dynamically construct virtual environments that reflect these obstructions and create a path that users can navigate through without hitting these physical objects. These are reflected on the images on the right of each of the pairings (210B, 220B, 230B).

For example, imagine you’re sat in your living room with your VR headset. In front of you, there is a table and a chair. Microsoft would use sensor data to construct a virtual environment to create a path that you could follow in the virtual world that would stop you from bumping into your table. This could be with a virtual wall appearing where your table is, while a virtual door appears where there there is empty space.

In the patent description, Microsoft mention the Microsoft Kinect as a possible source of sensor data. Or alternatively, the headset device itself could carry sensors for mapping out physical environments.

Why is this interesting?

Currently, for people to be able to enjoy VR to its fullest, they need to be in large empty rooms to be able to roam free without any physical obstructions. Companies like Sandbox VR tapped into this opportunity with physical locations for group VR experiences.

Otherwise, VR experiences in the home are constrained by not being able to allow for users to freely roam their physical environment. In turn, the VR experiences at home are relatively static. Either you move your virtual avatar with a controller rather than with your body, or the physical movement is restricted to a very small space (e.g. think of swinging your arms in Beat Saber).

Microsoft’s patent highlights the possibility of a deeper immersion – physical bodies freely roaming infinite virtual environments. Also, don’t forget the potentially huge commercial opportunities that arise from mapping the home environments of people around the world.

2. Square – video analysis of food service counters

This week, Square updated a fascinating Patent that hadn’t been touched in 4 years.

In this filing, Square describe a system for having a video camera record a food service counter of a restaurant, including the ingredient containers.

The camera would be connected to the point-of-sale device (i.e. the Square payment terminal). For each sale made, Square could then see which ingredients were required for each purchase.

With this information, Square mentions being able to do the following for restaurants:

• Ingredient layout recommendations: by seeing the most frequently selected ingredients, Square could make recommendations for how to change the layout of the ingredient counter in order to speed up service
• Menu recommendations: provide restaurants with a report of the most popular ingredient combinations to then inform menu changes
• Ingredient Replenishments: Square could track how quickly ingredients are being used up and in turn provide real-time data for when ingredients need to be replenished

This all sounds very interesting and useful.

Now, things get a little bit more strange with the following potential use cases:

• Evaluating effectiveness of food servers: for example, Square could track whether food servers are smiling during encounters with customers, and then offer suggestions to the restaurant to improve service.
• Evaluating customer satisfaction: Square could evaluate the faces of customers to understand customer satisfaction. This could then be correlated with the actions of food servers to understand which servers, or which actions, result in poor customer satisfaction.
• Customer demographic information: estimating gender and age of customers and seeing how this correlates with order choices. This could then feed into promotional and advertising choices.

The big caveat with patent filings is that there’s always an opportunism and imagination to these filings. Most of the time, they precede actual technological execution, and prudence encourages more use-cases being included than would actually be worked on.

That said, this filing is an interesting window into potential new revenue streams and services that Square could provide to its customers. By having a PoS terminal within the physical store, Square could in theory further embed itself into these restaurants, capture more data, and deeper understand the behaviour surrounding each transaction.

From a surveillance tech perspective, there are fascinating dynamics at play. Whatever data can be measured, will eventually be measured. By putting a video camera in the restaurant, Square suddenly opens up the possibility of tracking quality of employees, customer satisfaction and customer demographics, with everything synced to transaction data.

In this world, Square could help restaurants put a monetary value on good customer service – for better or worse.

3. Disney – virtual objects that hold your gaze

I like this Patent because it reminds me of how iterative magic is.

Disney want to make displays where digital characters could engage with specific members of an audience by tracking their eyes or by turning their heads and focusing on them.

The ambition is to make these displays more engaging and tailored to specific members of an audience. One can imagine these utilised in Disney’s amusement parks, advertisement displays or entertainment venues.

The system works by having a camera capture images of the audience, determining the ‘gaze view’ of a member of the audience, and then a computer rendering an image based on one individual’s gaze view.

For example, imagine a child walking by a display at Disneyland where Mickey Mouse welcomes guests . The display could detect one child’s gaze and then have Mickey Mouse looking into the eyes of that child, talking to them directly, following that child’s eyes as it walks by.

Then imagine the experience if it was an advertisement display where Disney want to get a child to buy a new toy. What impact would this gaze-sharing experience have on a child’s engagement with the ad and ultimately sales?

Such an experience would lift the experience of viewing characters on 2d-displays, making it seem more life-like and personalised. If it works well, one could imagine this being rolled out in more contexts, such as ads at bus stops and billboards. Or maybe even movies and games where individual characters can share a viewer’s gaze.