公开(公告)号：US20250061146A1

公开(公告)日：2025-02-20

申请号：US18802734

申请日：2024-08-13

Applicant: GOOGLE LLC

Inventor： Olivier Siegenthaler ,  Ágoston Weisz ,  Boris Bluntschli ,  Dan Banica ,  Kaan Ege Özgün ,  Daniel Mogoreanu ,  Filip Sladek

IPC: G06F16/532 ,  G06F40/40 ,  G06V10/80 ,  G06V20/50

Abstract: Implementations utilize an LLM to respond to queries comprising image data, such as multimodal queries that include both text and image data. A natural language processing system is extended such that when an image is provided, the natural language processing system invokes one or more auxiliary image processing models (e.g., visual query) and/or image search engines. The results, of invoking such model(s) and/or search engine(s), are collected into structured data signals related to the image. These signals form part of the conversation context and are used to extend the text prompt that is sent to the LLM. This allows the LLM to take the context into account when being used to process the user query, thereby enabling generation of an LLM reply that addresses relevant feature(s) of the image.