Google’s digital assistants may be doing more than just listening. 

The company is seeking to patent a way to predict automated assistant interactions using “fusion of visual and audio input.” Google’s tech aims for its digital assistants to detect and act upon specific commands without needing to be explicitly invoked. 

“There are various scenarios in which requiring users to utter long tail hot words before invoking an automated assistant to perform some action can be cumbersome,” Google said in the filing. 

In place of needing to say specific hot words or wake words like “hey Google,” this system would allow the user to invoke a digital assistant via “hot commands,” which are physical movements or voice activity that “represent an intent to interact with an automated assistant.” 

The system may catch these commands by listening passively to perform speech recognition “at time(s) other than immediately after the automated assistant is invoked,” Google noted. After listening, textual representations of a user’s speech are analyzed to figure out if the phrase is significant enough or said frequently enough to be considered a hot command. 

The process is similar for visual data: Google’s system would watch and analyze image data collected through vision sensors (such as digital cameras, passive infrared sensors, stereoscopic cameras and more) and analyze movements such as gestures, facial expressions or gaze direction, or with a “visual features module” to determine if the movements indicate a desired interaction.  

Finally, if the movements or phrases meet a certain threshold, they’re enrolled into Google’s system as a hot command, and used to perform specific actions when they’re said or done by the user. 

While this feature could show up in smart home or speaker devices, this could also be a part of Google’s plans for its smartphones. For one thing, its phones already come equipped with a camera to pick up the visual data that the patent mentions. While smartphone makers like Apple and Samsung are hard at work adding AI to their offerings, the latest Google Pixel already offers features using the company’s AI model Gemini.  

And Google has been plugging away at getting rid of hot words and wake phrases for a while. Its patent history includes inventions like non-wake word invocation of digital assistants using the context of your screen, digital assistants that collect physical data, and methods to train AI voice assistants via a global pot of training data.

This patent hints at a common theme in the world of AI development: Context is key, said Bob Rogers, PhD, co-founder of BeeKeeperAI and CEO of Oii.ai.  To get as much context as possible, AI models need to be able to handle multiple streams of data, whether it be text, audio or image. For AI to get users on a deeper level – whether that means anticipating their requests or reading their emotions – is going to take more than one stream of data. 

“The combination of video and audio definitely … creates a much better sense of context,” said Rogers. “In the last year, with the growth of multimodal foundational models, I think it’s starting to be kind of a no-brainer.” 

Enriching AI model’s data streams is crucial for voice assistant tech to move past wake words. But one major issue these systems face is false positives, said Rogers. In trying to catch as many signals as it can to build up a bevy of invocation-free commands, these systems are bound to act on requests that a user didn’t actually make. 

Another issue is privacy. While Google notes in this application that a lot of the data processing and storage is performed locally, it noted that cloud-based modules may be used for several of its components, including text-to-speech, speech-to-text, visual and audio feature modules and an “intent matcher.” And whether via your phone or your smart speaker, that means this system would be constantly siphoning up a lot of your data. 

“Just saying that they’re keeping some of the stream of data on the device is probably not sufficient to guarantee privacy,” Rogers said. “They’re probably giving themselves an out there, but they’re probably streaming quite a lot of data to the cloud.”

Nvidia wants to give its neural networks some depth perception. 

The company is seeking to patent techniques to “increase inference accuracy” with respect to autonomous machines. Nvidia’s method essentially allows neural networks to identify objects irrespective of their “size or perspective.” 

Neural networks cannot always identify objects if they are not “specifically trained to perform object detection for images having various objects with different size,” Nvidia said. “For example, objects that are far from said camera will appear smaller than other objects that are close to said camera.” 

Nvidia’s tech aims to overcome this problem by cutting images up into several parts, making it easier for the neural network to understand. The system determines how exactly to split up each image based on several factors: resolution of the object in the image, size required for processing by the neural network itself, “region of interest” within the image, and “user-defined parameters” (meaning, what the user wants the system to look for). 

The system would then localize and combine specific portions of the image for the neural network to get a clearer picture of what exactly it’s looking at, “resulting in better inference accuracy,” Nvidia said.  

To paint a picture, in the context of an autonomous vehicle, if it spots a hazard in the road from far away, a normal system may not detect it as a hazard at first, as it looks small on camera. But this system would be able to detect the hazard from farther away by separating it from the negative space around it. 

Nvidia’s wide-ranging patent puts this tech in the context of autonomous vehicles, as tech like this could help make self-driving cars’ decision-making skills more accurate. The company has a number of patent filings that aim to make the models in its vehicles sharper, including tech that automatically retrains image models from different perspectives, systems to perform fault detection, and ways to generate synthetic training data sets for visual AI. 

But similar to its previous patents in this realm, these all have far broader applications than simply driverless vehicles. All of these can be helpful tools for strengthening computer vision software generally to make object detection and task performance more accurate and efficient. This can be put to use in a host of different settings, from factory floors to retail environments to physical security. 

This also adds to the growing pile of AI software that Nvidia may be seeking to shore up as its AI chips growth continues at a breakneck pace.The company reported record quarterly revenue last week of $22.1 billion, up 265% year over year. Its data center revenue, which includes its AI chip sales, made up $18.4 billion of those earnings, up 409% from last year. The news shot Nvidia’s market cap up by $277 billion on Thursday, and it briefly hit $2 trillion on Friday.

Demand for its chips have reached such a fever pitch that the company has to carefully consider how to “fairly” dole them out, CEO Jensen Huang said on the company’s earnings call. “We do the best we can to allocate fairly, and to avoid allocating unnecessarily.” 

While the company’s chip sales are what continue to make headlines, its software ecosystem is nothing to sneeze at, especially as its patent filings continue to cover more ground. Nvidia’s AI software research could help it expand its horizons as the AI industry clambers to get a hold of its hardware — and potentially looks for cheaper alternatives.

eBay may want to use blockchain to prevent its users from getting scammed. 

The company is seeking to patent “blockchain-based authentication and authorization.” eBay’s system essentially aims to bring the transparency and decentralization that blockchain provides to the process of authenticating its sellers.

“Authentication and authorization of users with respect to permissions granted by a central authority may be opaque to other users,” eBay said. “There may be situations in which the user is reluctant to extend trust to another user based only on assurances provided by a central authority.”

eBay’s system would give user’s ownership or control over a specific blockchain address using a private key. Once a user account is linked to a blockchain address, it’s recorded on the blockchain itself, and can be used for authentication. On the authorization side, ownership of a blockchain address would allow the user to “sign for transactions,” eBay said, essentially serving as a digital signature. 

For authorization of these transactions, eBay’s system may come with a few stipulations. For one, a user may or may not be authorized to buy something or participate in certain auctions based on their previous transactions recorded on the blockchain. This system would record things like purchase history, sales history and payment history. For example, if a user wants to bid on a high-value item, they may be prohibited from doing so based on the amount of funds available in their cryptocurrency wallet. 

This system may also use the blockchain to track a user’s “reputational score,” which could also determine whether they could buy, bid on or sell items. This score would fluctuate based on transaction behavior recorded on the blockchain. 

Given its nature as a peer-to-peer marketplace, eBay researching ways to take on fraud using blockchain makes sense. Buyers and sellers need to be able to trust one another, and eBay has run into criticism in the past for fraud and counterfeits. 

And late last month, the company reached a settlement with the US Department of Justice  regarding the sale of pill presses and encapsulating machines on its site. The settlement requires eBay to pay $59 million and implement stricter processes for monitoring and reporting listings that violate its policies, though eBay denied the DOJ’s allegations that it violated any laws. 

While the company is looking toward AI to verify the authenticity of its listings, the inherent features of blockchain may actually be just what eBay needs to keep the sellers themselves honest. eBay noted in the patent that with a blockchain-based reputation score, “bad actors can be identified and the immutability of records contained in the blockchain will make it difficult for those actors to distance themselves from their past behavior.” 

Plus, this isn’t eBay’s first blockchain rodeo. The company previously sought to patent logistics tech that relied on blockchain to secure private information on shipping labels and methods to “fingerprint” physical items using NFTs.

And in mid-2022, the ecommerce firm invested heavily in blockchain tech with the acquisition of NFT marketplace KnownOrigin. However, since the acquisition, blockchain has had a fall from grace, with eBay reportedly laying off 30% of KnownOrigin’s staff and backing away from its plans for NFTs. 

With patents like these, the company may be looking for a way to put the underlying benefits of blockchain to good use amid rocky waters for the industry.