公开(公告)号：US12125317B2

公开(公告)日：2024-10-22

申请号：US17539652

申请日：2021-12-01

Applicant: ADOBE INC.

Inventor： Jiyoung Lee ,  Justin Jonathan Salamon ,  Dingzeyu Li

IPC: G06V40/20 ,  G06N3/045 ,  G06N3/08 ,  G06V10/82 ,  G06V20/40

CPC classification number: G06V40/20 ,  G06N3/045 ,  G06N3/08 ,  G06V10/82 ,  G06V20/41 ,  G06V20/46 ,  G06V20/49

Abstract: A method for detecting a cue (e.g., a visual cue or a visual cue combined with an audible cue) occurring together in an input video includes: presenting a user interface to record an example video of a user performing an act including the cue; determining a part of the example video where the cue occurs; applying a feature of the part to a neural network to generate a positive embedding; dividing the input video into a plurality of chunks and applying a feature of each chunk to the neural network to generate a plurality of negative embeddings; applying a feature of a given one of the chunks to the neural network to output a query embedding; and determining whether the cue occurs in the input video from the query embedding, the positive embedding, and the negative embeddings.

公开(公告)号：US11922695B2

公开(公告)日：2024-03-05

申请号：US17805076

申请日：2022-06-02

Applicant: ADOBE INC.

Inventor： Hijung Shin ,  Xue Bai ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Joy Oakyung Kim ,  Seth Walker

IPC: G06V20/40 ,  G06F18/231 ,  G10L25/78 ,  G11B27/00 ,  G11B27/19 ,  G10L25/18

CPC classification number: G06V20/49 ,  G06F18/231 ,  G06V20/41 ,  G06V20/46 ,  G10L25/78 ,  G11B27/002 ,  G11B27/19 ,  G06V20/44

Abstract: Embodiments are directed to segmentation and hierarchical clustering of video. In an example implementation, a video is ingested to generate a multi-level hierarchical segmentation of the video. In some embodiments, the finest level identifies a smallest interaction unit of the video—semantically defined video segments of unequal duration called clip atoms. Clip atom boundaries are detected in various ways. For example, speech boundaries are detected from audio of the video, and scene boundaries are detected from video frames of the video. The detected boundaries are used to define the clip atoms, which are hierarchically clustered to form a multi-level hierarchical representation of the video. In some cases, the hierarchical segmentation identifies a static, pre-computed, hierarchical set of video segments, where each level of the hierarchical segmentation identifies a complete set (i.e., covering the entire range of the video) of disjoint (i.e., non-overlapping) video segments with a corresponding level of granularity.

公开(公告)号：US11893794B2

公开(公告)日：2024-02-06

申请号：US17805080

申请日：2022-06-02

Applicant: ADOBE INC.

Inventor： Hijung Shin ,  Xue Bai ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Joy Oakyung Kim ,  Seth Walker

IPC: G06F18/231 ,  G06V20/40 ,  G11B27/19 ,  G11B27/00 ,  G10L25/78 ,  G06V20/00 ,  G06V20/70

CPC classification number: G06V20/49 ,  G06F18/231 ,  G06V20/41 ,  G06V20/46 ,  G10L25/78 ,  G11B27/002 ,  G11B27/19 ,  G06V20/44

Abstract: Embodiments are directed to segmentation and hierarchical clustering of video. In an example implementation, a video is ingested to generate a multi-level hierarchical segmentation of the video. In some embodiments, the finest level identifies a smallest interaction unit of the video—semantically defined video segments of unequal duration called clip atoms. Clip atom boundaries are detected in various ways. For example, speech boundaries are detected from audio of the video, and scene boundaries are detected from video frames of the video. The detected boundaries are used to define the clip atoms, which are hierarchically clustered to form a multi-level hierarchical representation of the video. In some cases, the hierarchical segmentation identifies a static, pre-computed, hierarchical set of video segments, where each level of the hierarchical segmentation identifies a complete set (i.e., covering the entire range of the video) of disjoint (i.e., non-overlapping) video segments with a corresponding level of granularity.

公开(公告)号：US11887371B2

公开(公告)日：2024-01-30

申请号：US17330718

申请日：2021-05-26

Applicant: ADOBE INC.

Inventor： Seth Walker ,  Hijung Shin ,  Cristin Ailidh Fraser ,  Aseem Agarwala ,  Lubomira Dontcheva ,  Joel Richard Brandt ,  Jovan Popović ,  Joy Oakyung Kim ,  Justin Salamon ,  Jui-hsien Wang ,  Timothy Jeewun Ganter ,  Xue Bai ,  Dingzeyu Li

IPC: G11B27/00 ,  G11B27/031 ,  G11B27/34 ,  G06V20/40 ,  G11B27/10 ,  G06V10/75 ,  G06V40/16 ,  G06V10/82

CPC classification number: G06V20/49 ,  G06V10/751 ,  G06V20/46 ,  G06V40/161 ,  G11B27/10

Abstract: Embodiments are directed to a thumbnail segmentation that defines the locations on a video timeline where thumbnails are displayed. Candidate thumbnail locations are determined from boundaries of feature ranges of the video indicating when instances of detected features are present in the video. In some embodiments, candidate thumbnail separations are penalized for being separated by less than a minimum duration corresponding to a minimum pixel separation (e.g., the width of a thumbnail) between consecutive thumbnail locations on a video timeline. The thumbnail segmentation is computed by solving a shortest path problem through a graph that models different thumbnail locations and separations. As such, a video timeline is displayed with thumbnails at locations on the timeline defined by the thumbnail segmentation, with each thumbnail depicting a portion of the video associated with the thumbnail location.

公开(公告)号：US11880408B2

公开(公告)日：2024-01-23

申请号：US17017370

申请日：2020-09-10

Applicant: ADOBE INC.

Inventor： Seth Walker ,  Joy Oakyung Kim ,  Morgan Nicole Evans ,  Najika Skyler Halsema Yoo ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Hijung Shin ,  Xue Bai

IPC: G06F16/75 ,  G06F16/738 ,  G06T13/80 ,  G06F3/0482 ,  G06F16/74 ,  G06F16/735 ,  G06F3/0484

CPC classification number: G06F16/738 ,  G06F3/0482 ,  G06F3/0484 ,  G06F16/735 ,  G06F16/743 ,  G06F16/75 ,  G06T13/80

Abstract: Embodiments are directed to techniques for interacting with a hierarchical video segmentation by performing a metadata search. Generally, various types of metadata can be extracted from a video, such as a transcript of audio, keywords from the transcript, content or action tags visually extracted from video frames, and log event tags extracted from an associated temporal log. The extracted metadata is segmented into metadata segments and associated with corresponding video segments defined by a hierarchical video segmentation. As such, a metadata search can be performed to identify matching metadata segments and corresponding matching video segments defined by a particular level of the hierarchical segmentation. Matching metadata segments are emphasized in a composite list of the extracted metadata, and matching video segments are emphasized on the video timeline. Navigating to a different level of the hierarchy transforms the search results into corresponding coarser or finer segments defined by the level.

公开(公告)号：US11875568B2

公开(公告)日：2024-01-16

申请号：US17805076

申请日：2022-06-02

Applicant: ADOBE INC.

Inventor： Hijung Shin ,  Xue Bai ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Joy Oakyung Kim ,  Seth Walker

IPC: G06V20/40 ,  G11B27/00 ,  G11B27/19 ,  G10L25/78 ,  G06F18/231 ,  G10L25/18

CPC classification number: G06V20/49 ,  G06F18/231 ,  G06V20/41 ,  G06V20/46 ,  G10L25/78 ,  G11B27/002 ,  G11B27/19 ,  G06V20/44

Abstract: Embodiments are directed to segmentation and hierarchical clustering of video. In an example implementation, a video is ingested to generate a multi-level hierarchical segmentation of the video. In some embodiments, the finest level identifies a smallest interaction unit of the video—semantically defined video segments of unequal duration called clip atoms. Clip atom boundaries are detected in various ways. For example, speech boundaries are detected from audio of the video, and scene boundaries are detected from video frames of the video. The detected boundaries are used to define the clip atoms, which are hierarchically clustered to form a multi-level hierarchical representation of the video. In some cases, the hierarchical segmentation identifies a static, pre-computed, hierarchical set of video segments, where each level of the hierarchical segmentation identifies a complete set (i.e., covering the entire range of the video) of disjoint (i.e., non-overlapping) video segments with a corresponding level of granularity.

公开(公告)号：US11822602B2

公开(公告)日：2023-11-21

申请号：US17017370

申请日：2020-09-10

Applicant: ADOBE INC.

Inventor： Seth Walker ,  Joy Oakyung Kim ,  Morgan Nicole Evans ,  Najika Skyler Halsema Yoo ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Hijung Shin ,  Xue Bai

IPC: G06F16/75 ,  G06F16/738 ,  G06T13/80 ,  G06F3/0482 ,  G06F16/74 ,  G06F16/735 ,  G06F3/0484

CPC classification number: G06F16/738 ,  G06F3/0482 ,  G06F3/0484 ,  G06F16/735 ,  G06F16/743 ,  G06F16/75 ,  G06T13/80

Abstract: Embodiments are directed to techniques for interacting with a hierarchical video segmentation by performing a metadata search. Generally, various types of metadata can be extracted from a video, such as a transcript of audio, keywords from the transcript, content or action tags visually extracted from video frames, and log event tags extracted from an associated temporal log. The extracted metadata is segmented into metadata segments and associated with corresponding video segments defined by a hierarchical video segmentation. As such, a metadata search can be performed to identify matching metadata segments and corresponding matching video segments defined by a particular level of the hierarchical segmentation. Matching metadata segments are emphasized in a composite list of the extracted metadata, and matching video segments are emphasized on the video timeline. Navigating to a different level of the hierarchy transforms the search results into corresponding coarser or finer segments defined by the level.

公开(公告)号：US11631434B2

公开(公告)日：2023-04-18

申请号：US17017362

申请日：2020-09-10

Applicant: ADOBE INC.

Inventor： Seth Walker ,  Joy Oakyung Kim ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popović ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Hijung Shin ,  Xue Bai

IPC: G11B27/034 ,  G11B27/34 ,  G06F16/783 ,  G06F3/0485 ,  G06F16/75 ,  G06V20/40

Abstract: Embodiments are directed to techniques for interacting with a hierarchical video segmentation. In some embodiments, the finest level of the hierarchical segmentation identifies the smallest interaction unit of a video—semantically defined video segments of unequal duration called clip atoms. Each level of the hierarchical segmentation clusters the clip atoms with a corresponding degree of granularity into a corresponding set of video segments. A presented video timeline is segmented based on one of the levels, and one or more segments are selected through interactions with the video timeline (e.g., clicks, drags), by performing a metadata search, or through selection of corresponding metadata segments from a metadata panel. Navigating to a different level of the hierarchy transforms the selection into corresponding coarser or finer video segments defined by the level. Any operation can be performed on selected video segments, including playing back, trimming, or editing.

公开(公告)号：US11417041B2

公开(公告)日：2022-08-16

申请号：US16788551

申请日：2020-02-12

Applicant: ADOBE INC.

Inventor： Dingzeyu Li ,  Yang Zhou ,  Jose Ignacio Echevarria Vallespi ,  Elya Shechtman

IPC: G06T13/20 ,  G06T13/40 ,  G06T17/20

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media for generating an animation of a talking head from an input audio signal of speech and a representation (such as a static image) of a head to animate. Generally, a neural network can learn to predict a set of 3D facial landmarks that can be used to drive the animation. In some embodiments, the neural network can learn to detect different speaking styles in the input speech and account for the different speaking styles when predicting the 3D facial landmarks. Generally, template 3D facial landmarks can be identified or extracted from the input image or other representation of the head, and the template 3D facial landmarks can be used with successive windows of audio from the input speech to predict 3D facial landmarks and generate a corresponding animation with plausible 3D effects.

公开(公告)号：US20220076705A1

公开(公告)日：2022-03-10

申请号：US17017362

申请日：2020-09-10

Applicant: ADOBE INC.

Inventor： Seth Walker ,  Joy Oakyung Kim ,  Aseem Agarwala ,  Joel R. Brandt ,  Jovan Popovic ,  Lubomira Dontcheva ,  Dingzeyu Li ,  Hijung Shin ,  Xue Bai

IPC: G11B27/034 ,  G11B27/34 ,  G06F16/783 ,  G06F16/75 ,  G06F3/0485 ,  G06K9/00

Abstract: Embodiments are directed to techniques for interacting with a hierarchical video segmentation. In some embodiments, the finest level of the hierarchical segmentation identifies the smallest interaction unit of a video—semantically defined video segments of unequal duration called clip atoms. Each level of the hierarchical segmentation clusters the clip atoms with a corresponding degree of granularity into a corresponding set of video segments. A presented video timeline is segmented based on one of the levels, and one or more segments are selected through interactions with the video timeline (e.g., clicks, drags), by performing a metadata search, or through selection of corresponding metadata segments from a metadata panel. Navigating to a different level of the hierarchy transforms the selection into corresponding coarser or finer video segments defined by the level. Any operation can be performed on selected video segments, including playing back, trimming, or editing.