公开(公告)号：US11741350B2

公开(公告)日：2023-08-29

申请号：US16698461

申请日：2019-11-27

Applicant: Amazon Technologies, Inc.

Inventor： Jeffrey T. Huynh ,  Ron Diamant ,  Hongbin Zheng ,  Yizhi Liu ,  Animesh Jain ,  Yida Wang ,  Vinod Sharma ,  Richard John Heaton ,  Randy Renfu Huang ,  Sundeep Amirineni ,  Drazen Borkovic

IPC: G06N3/063 ,  G06N3/04 ,  G06N3/08

CPC classification number: G06N3/063 ,  G06N3/04

Abstract: A computer-implemented method includes receiving a neural network model for implementation using a processing element array, where the neural network model includes a convolution operation on a set of input feature maps and a set of filters. The method also includes determining, based on the neural network model, that the convolution operation utilizes less than a threshold number of rows in the processing element array for applying a set of filter elements to the set of input feature maps, where the set of filter elements includes one filter element in each filter of the set of filters. The method further includes generating, for the convolution operation and based on the neural network model, a first instruction and a second instruction for execution by respective rows in the processing element array, where the first instruction and the second instruction use different filter elements of a filter in the set of filters.

公开(公告)号：US11175919B1

公开(公告)日：2021-11-16

申请号：US16219610

申请日：2018-12-13

Applicant: Amazon Technologies, Inc.

Inventor： Ilya Minkin ,  Ron Diamant ,  Drazen Borkovic ,  Jindrich Zejda ,  Dana Michelle Vantrease

IPC: G06F9/30 ,  G06F9/35 ,  G06F13/28 ,  G06F9/38 ,  G06F9/52 ,  G06N3/06

Abstract: Integrated circuit devices and methods for synchronizing execution of program code for multiple concurrently operating execution engines of the integrated circuit devices are provided. In some cases, one execution engine of an integrated circuit device may be dependent on the operation of another execution engine of the integrated circuit device. To synchronize the execution engines around the dependency, a first execution engine may execute an instruction to set a value in a register while a second execution engine may execute an instruction to wait for a condition associated with the register value.

公开(公告)号：US11061654B1

公开(公告)日：2021-07-13

申请号：US16217797

申请日：2018-12-12

Applicant: Amazon Technologies, Inc.

Inventor： Drazen Borkovic ,  Jindrich Zejda ,  Taemin Kim ,  Ron Diamant

IPC: G06F9/44 ,  G06F8/41 ,  G06F9/30 ,  G06N3/02 ,  G06F12/1081

Abstract: Provided are systems and methods for synchronizing program code execution for a plurality of execution engines in an integrated circuit device. In some cases, the operation of one execution engine may be dependent on the operation of another execution engine. To accommodate this dependency, the instructions for the first execution engine can include a set-event instruction and the instructions for the second execution engine can include a wait-on-event instruction. The wait-on-event instruction can cause the second execution engine to wait for the first execution engine to reach the set-event instruction. In this way, the two execution engines can be synchronized around the data or resource dependency.

公开(公告)号：US20210158131A1

公开(公告)日：2021-05-27

申请号：US16698236

申请日：2019-11-27

Applicant: Amazon Technologies, Inc.

Inventor： Animesh Jain ,  Yizhi Liu ,  Hongbin Zheng ,  Jeffrey T. Huynh ,  Haichen Li ,  Drazen Borkovic ,  Jindrich Zejda ,  Richard John Heaton ,  Randy Renfu Huang ,  Zhi Chen ,  Yida Wang

IPC: G06N3/063 ,  G06N3/04

Abstract: Methods and apparatuses for hierarchical partitioning of operators of a neural network for execution on an acceleration engine are provided. Neural networks are built in machine learning frameworks using neural network operators. The neural network operators are compiled into executable code for the acceleration engine. Development of new framework-level operators can exceed the capability to map the newly developed framework-level operators onto the acceleration engine. To enable neural networks to be executed on an acceleration engine, hierarchical partitioning can be used to partition the operators of the neural network. The hierarchical partitioning can identify operators that are supported by a compiler for execution on the acceleration engine, operators to be compiled for execution on a host processor, and operators to be executed on the machine learning framework.

公开(公告)号：US11016775B2

公开(公告)日：2021-05-25

申请号：US16453478

申请日：2019-06-26

Applicant: Amazon Technologies, Inc.

Inventor： Jeffrey T. Huynh ,  Drazen Borkovic ,  Jindrich Zejda ,  Randy Renfu Huang ,  Ron Diamant

IPC: G06F8/00 ,  G06F9/38 ,  G06F9/50 ,  G06N3/04 ,  G06N3/08

Abstract: Techniques are disclosed for reordering operations of a neural network to improve runtime efficiency. In some examples, a compiler receives a description of the neural network comprising a plurality of operations. The compiler may determine which execution engine of a plurality of execution engines is to perform each of the plurality of operations. The compiler may determine an order of performance associated with the plurality of operations. The compiler may identify a runtime inefficiency based on the order of performance and a hardware usage for each of the plurality of operations. An operation may be reordered to reduce the runtime inefficiency. Instructions may be compiled based on the plurality of operations, which include the reordered operation.

公开(公告)号：US10922146B1

公开(公告)日：2021-02-16

申请号：US16219530

申请日：2018-12-13

Applicant: Amazon Technologies, Inc.

Inventor： Ilya Minkin ,  Ron Diamant ,  Drazen Borkovic ,  Jindrich Zejda ,  Dana Michelle Vantrease

IPC: G06F9/46 ,  G06F9/52 ,  G06F8/41 ,  G06F9/30 ,  G06N3/063

Abstract: Systems and methods are provided for synchronizing execution of program code for an integrated circuit device having multiple concurrently operating execution engines, where the operation of one execution engine may be dependent on the operation of another execution engine. Data or resource dependencies may be accommodated with a Set instruction to cause a first execution engine to set a register value and a Wait instruction to cause a second execution engine to wait for a condition associate with the register value. Concurrently operation of the execution engines may thus be synchronized.

公开(公告)号：US12198041B2

公开(公告)日：2025-01-14

申请号：US18352768

申请日：2023-07-14

Applicant: Amazon Technologies, Inc.

Inventor： Jeffrey T. Huynh ,  Ron Diamant ,  Hongbin Zheng ,  Yizhi Liu ,  Animesh Jain ,  Yida Wang ,  Vinod Sharma ,  Richard John Heaton ,  Randy Renfu Huang ,  Sundeep Amirineni ,  Drazen Borkovic

IPC: G06N3/063 ,  G06N3/04 ,  G06N3/045 ,  G06N3/08

Abstract: Generating instructions for programming a processing element array to implement a convolution operation can include determining that the convolution operation under-utilizes the processing element array. The convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices. Each filter comprises a weight matrix. Each input matrix is assigned to a respective row in the processing element array. Under-utilization can be determined through detecting that less than a threshold number of rows would be used concurrently. In response to determining that the convolution operation under-utilizes the processing element array, instructions can be added for modifying the convolution operation to increase the number of rows used concurrently. The added instructions are executable to cause at least one input matrix to be processed in parallel across more rows compared to processing without modifying the convolution operation.

公开(公告)号：US11847507B1

公开(公告)日：2023-12-19

申请号：US17247180

申请日：2020-12-02

Applicant: Amazon Technologies, Inc.

Inventor： Drazen Borkovic

IPC: G06F9/52 ,  G06F13/28

CPC classification number: G06F9/52 ,  G06F13/28

Abstract: Two or more semaphores can be used per queue for synchronization of direct memory access (DMA) transfers between a DMA engine and various computational engines by alternating the semaphores across sequential sets of consecutive DMA transfers in the queue. The DMA engine can increment a first semaphore after performing each DMA transfer of a first set of consecutive DMA transfers and a second semaphore after performing each DMA transfer of a second set of consecutive DMA transfers that is after the first set of consecutive DMA transfers in the queue. Each semaphore can be reset when all the computational engines that are dependent on the respective set of consecutive DMA transfers are done waiting on the given semaphore before performing respective operations. After reset, the first semaphore or the second semaphore can be reused for the next set of consecutive DMA transfers in the queue.

公开(公告)号：US11782706B1

公开(公告)日：2023-10-10

申请号：US17361992

申请日：2021-06-29

Applicant: Amazon Technologies, Inc.

Inventor： Ron Diamant ,  Hongbin Zheng ,  Drazen Borkovic ,  Haichen Li

IPC: G06F8/41 ,  G06N3/063 ,  G06F9/30 ,  G06F7/548 ,  G06N3/04

CPC classification number: G06F9/3001 ,  G06F7/548 ,  G06F8/433 ,  G06F8/443 ,  G06N3/04 ,  G06N3/063

Abstract: In one example, a method comprises: receiving input codes, wherein the input codes represent a computational dataflow graph; traversing the computational dataflow graph to identify single-entry-single-exit (SESE) subgraphs of the computational dataflow graph, wherein each SESE subgraph has a sequence of nodes comprising a root node and a child node and representing a sequence of element-wise operators, wherein the root node receives a single input tensor, and wherein the child node outputs a single output tensor; determining a merged operator for each SESE subgraph; and generating executable instructions for the computational dataflow graph to be executed by a hardware accelerator having a first execution unit and a second execution unit, wherein the executable instructions comprise first executable instructions for the merged operators targeted at the first execution unit, and second executable instructions for other operators of the computational dataflow graph targeted at the second execution unit.

公开(公告)号：US11567778B2

公开(公告)日：2023-01-31

申请号：US17243415

申请日：2021-04-28

Applicant: Amazon Technologies, Inc.

Inventor： Jeffrey T. Huynh ,  Drazen Borkovic ,  Jindrich Zejda ,  Randy Renfu Huang ,  Ron Diamant

IPC: G06F9/44 ,  G06F9/38 ,  G06F9/50 ,  G06N3/04 ,  G06N3/08

Abstract: Techniques are disclosed for reordering operations of a neural network to improve runtime efficiency. In some examples, a compiler receives a description of the neural network comprising a plurality of operations. The compiler may determine which execution engine of a plurality of execution engines is to perform each of the plurality of operations. The compiler may determine an order of performance associated with the plurality of operations. The compiler may identify a runtime inefficiency based on the order of performance and a hardware usage for each of the plurality of operations. An operation may be reordered to reduce the runtime inefficiency. Instructions may be compiled based on the plurality of operations, which include the reordered operation.