公开(公告)号：US20250005452A1

公开(公告)日：2025-01-02

申请号：US18708948

申请日：2023-01-24

Applicant: QUALCOMM Incorporated

Inventor： Markus NAGEL ,  Marios FOURNARAKIS ,  Tijmen Pieter Frederik BLANKEVOORT ,  Yelysei BONDARENKO

IPC: G06N20/00

Abstract: Certain aspects of the present disclosure provide techniques and apparatus for mitigating weight oscillation during quantization-aware training. In one example, a method includes identifying oscillation of a parameter of a machine learning model during quantization-aware training of the machine learning model, and applying an oscillation mitigation procedure during the quantization-aware training of the machine learning model in response to identifying the oscillation, the oscillation mitigation procedure comprising at least one of oscillation dampening or parameter freezing.

公开(公告)号：US20240386239A1

公开(公告)日：2024-11-21

申请号：US18482196

申请日：2023-10-06

Applicant: QUALCOMM Incorporated

Inventor： Yelysei BONDARENKO ,  Markus NAGEL ,  Tijmen Pieter Frederik BLANKEVOORT

IPC: G06N3/04

Abstract: Certain aspects of the present disclosure provide techniques and apparatus for processing data using a transformer neural network. The method generally includes receiving an input for processing using a transformer neural network. An attention output is generated in the transformer neural network. Generally, the attention output may be generated such that outlier values for the attention output are attenuated in the transformer neural network. An output of the transformer neural network is generated based on the generated attention output.

公开(公告)号：US20230139347A1

公开(公告)日：2023-05-04

申请号：US17976683

申请日：2022-10-28

Applicant: QUALCOMM Incorporated

Inventor： Yelysei BONDARENKO ,  Markus NAGEL ,  Tijmen Pieter Frederik BLANKEVOORT

IPC: G06F40/40 ,  G06N3/04 ,  G06N3/08

Abstract: A processor-implemented method for providing per-embedding-group activation quantization includes receiving sequential data at a first layer of a transformer neural network. The sequential data is processed via the first layer of the transformer neural network to generate an activation tensor. The activation tensor is split into multiple groups of embeddings. Each of the embeddings groups has a different set of quantization parameters. Each of the embedding groups is quantized separately based on the corresponding quantization parameters of the different set of quantization parameters. The quantized embedding groups are multiplied with a set of weights to generate an output.