Google’s VR-controlled robots, Disney’s drone shows & more

1. Google – controlling robots remotely

Google is thinking about using humans to remotely train robots to perform complex or unfamiliar tasks on a production line.

Robots are usually pre-programmed for very specific tasks. For instance, if a robot has to repeatedly apply a fastener to a specific component on an assembly line, the instructions would be: grasp a component from one fixed location, apply it to another fixed location, and repeat.

However, in scenarios where there are a lot of components, or some new components that a robot hasn’t seen before, a significant engineering effort is usually required in pre-programming the robot to make it fully productive.

To quickly train robots in these circumstances, Google is thinking about having people remotely control a robot to perform a task on the assembly line. This could be through people using VR headsets to share the same ‘view’ as a robot, and then controlling the robot’s hands with VR gloves or hand controllers.

Based on this ‘human input’. a machine learning model can then be trained in order to predict what the robot needs to do in order to complete a task successfully. In turn, the robot will quickly be able to work autonomously, or at the very least, with minor human intervention to ensure full productivity on the assembly line.

This filing is interesting for a whole host of reasons. The one that stands out to me is that we have an example of how VR, gaming and remote work can come together. One opportunity I’ve been thinking about recently is the idea of using game-like interfaces to make training machine learning models fun, quick and cheap. These games could be built to be extremely entertaining, financially rewarding, and have real-world utility.

One future prediction is that the worlds of enterprise, gaming and AI will converge in fascinating ways, and this filing is a hint of this possibility.

2. Microsoft – mapping IoT sensor data in Mixed Reality

Okay, this latest Microsoft patent filing is a bit of a trip but let’s try to break it down.

Our homes and workplaces are increasingly filled with IoT devices (Internet Of Things) which are used to connect vehicles, homes, appliances, or any other type of electronic device to the Internet. One example of an IoT device is a smart thermostat that automatically monitors and controls the temperatures of our homes.

In this filing, Microsoft wants to use VR headsets (or AR devices) to visualise the sensor data being picked up by all of these IoT devices in our local environment.

The main problem Microsoft is trying to solve is helping people easily track where IoT devices are located and what conditions they are monitoring. For example, the temperature conditions of a room can be monitored, but we may not know which room is specifically being monitored. Given how portable, small and ubiquitous IoT devices are becoming, Microsoft foresees a problem of managing all of these devices.

For example, in the above image, there is a temperature sensor that is monitoring conditions in the kitchen. In the headset, the temperature data of the device is visualised to show that the area close to the pot is hotter than the rest of the environment.

I think this filing is interesting in that it describes augmenting human senses by leveraging all of the sensors and data captured by IoT devices. For example, imagine if you walk into a building, your headset device could essentially allow you to ‘see’ another room that you’re not in by connecting to the temperature sensors in that room.

Where is this useful? If there was a fire in a building, perhaps soon we’ll be sending in robots that are remotely controlled by humans. Using data from IoT sensors, the robot could show the human controller where the fire source is located and if there are any humans in the building. Using a VR headset, the human controller could then navigate the robot to rescue anyone in danger or put out the fire.

This is of course speculative, but the idea of VR headsets / AR glasses syncing up with sensor data from IoT devices to augment our senses is something that will undoubtedly happen.

3. Disney – drones for entertainment shows

Disney is thinking about building a platform for drones to provide entertainment at facilities such as theme parks, sports stadiums and cruise ships.

Generally, sky-based entertainment shows have relied on systems that are positioned on the ground. For instance, the Disney fireworks show synchronises the launching of the rockets with a soundtrack.

Recently, there have been attempts at using drones to provide sky-based entertainment – in fact, London brought in 2021 with a drone show.

Drone shows tend to work by following a pre-programmed set of movements. However, their effectiveness in creating high quality shows that appear synchronous is limited. For instance, weather conditions might mean that the drones deviate in time or space from where they ought to be. Moreover, it is difficult to synchronise the drones with actors or ground components for more complex shows.

Disney is trying to solve this with a platform that will allow drones to create dynamically coordinated shows. This means that the drones’ position and orientation may change for each show in a way that ensures the best performance. The drones will also communicate with a ground-based control system so that they can respond to trigger signals e.g. when the soundtrack gets to a specific part, the drones must do X, taking into account any deviations that may have happened during the show.

To add to the magic, Disney describes the drones as having a structural frame to support an outer covering. With this, the drones can be made to resemble characters or objects from a movie, while being lightweight enough to still fly in the air.

While Disney has partnered with Intel in the past to put on a drone light show, it looks like the company is now looking to build its own in-house technology that can be used to create better, more cohesive, drone-based entertainment.

With Walt Disney’s empire spanning across Disney, ESPN, Marvel and LucasFilms, maybe we’ll soon start seeing drone-based entertainment using characters from their IP portfolio, as well as new types of entertainment during sports breaks at stadiums.