Dynamic partitioning techniques for data streams

    公开(公告)号:US10691716B2

    公开(公告)日:2020-06-23

    申请号:US15665103

    申请日:2017-07-31

    Abstract: A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.

    Idempotent processing of data streams

    公开(公告)号:US10601881B1

    公开(公告)日:2020-03-24

    申请号:US14875201

    申请日:2015-10-05

    Inventor: Gaurav D. Ghare

    Abstract: Idempotent processing of data may be implemented for data records retrieved from a data stream. A data stream may receive data records as input and distribute the ingestion, storage, and processing of the data records amongst one or more partitions of the data stream. Partition metadata may be maintained which includes checkpoint metadata for retrieving, processing, and sending data records in the data stream to a specified destination. When assigned a partition for processing, checkpoint metadata for partition may be accessed to determine whether a pending checkpoint for the partition exists. If not pending checkpoint exists, new data records may be retrieved, processed, and sent from the partition of the data stream to a specified destination. If a checkpoint is pending, then the data records identified by the checkpoint metadata as pending may be retrieved, processed, and sent to the specified destination.

    Managed function execution for processing data streams in real time

    公开(公告)号:US10122788B2

    公开(公告)日:2018-11-06

    申请号:US15084347

    申请日:2016-03-29

    Abstract: Managed function execution for processing data streams in real time may be. A function that describes one or more operations to be performed with respect to one or more data streams may be received via programmatic interface for a managed stream processing system. Stream processing nodes capable of applying the function may be determined and execution of the one or more operations may be initiated at the stream processing nodes as data records of the data stream are received. Results of the application of the processing function may be provided to one or more destinations specified for the function. Performance metrics may also be collected for the execution of the function and provided to a client that submitted the function.

    Dynamic management of data stream processing

    公开(公告)号:US09858124B1

    公开(公告)日:2018-01-02

    申请号:US14875194

    申请日:2015-10-05

    Inventor: Gaurav D. Ghare

    CPC classification number: G06F9/5083

    Abstract: A stream management system may implement dynamic management of a data stream. Utilization data of different partitions of a data stream may be tracked. When routing a data record received at the stream management system, a partition may be dynamically identified for the data recorded. The data record may then be directed to the identified partition. Other management operations, such as repartitioning the data stream or reassigning resources for processing data records in the data stream may be performed based on the utilization data tracked for the partitions.

    Dynamic partitioning techniques for data streams

    公开(公告)号:US09720989B2

    公开(公告)日:2017-08-01

    申请号:US14077171

    申请日:2013-11-11

    CPC classification number: G06F17/30569 G06F17/30

    Abstract: A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.

    USING CONFIGURED APPLICATION INFORMATION TO CONTROL USE OF INVOCABLE SERVICES
    17.
    发明申请
    USING CONFIGURED APPLICATION INFORMATION TO CONTROL USE OF INVOCABLE SERVICES 审中-公开
    使用配置的应用信息来控制可用服务的使用

    公开(公告)号:US20150235191A1

    公开(公告)日:2015-08-20

    申请号:US14701377

    申请日:2015-04-30

    Abstract: Techniques are described for facilitating use of invocable services by applications in a configurable manner. In at least some situations, the invocable services are Web services or other network-accessible services that are made available by providers of the services for use by others in exchange for fees defined by the service providers. The described techniques facilitate use of such invocable services by applications in a manner configured by the application providers and the service providers, including to track use of third-party invocable services by applications on behalf of end users and to allocate fees that are charged end users between the applications and the services as configured by the providers of the applications and services. In some situations, the configured pricing terms for a service specify fees for end users that differ in one or more ways from the defined fees charged by the provider of that service.

    Abstract translation: 描述了以可配置的方式促进应用使用可调用服务的技术。 在至少一些情况下,可调用的服务是Web服务或其他网络可访问的服务,这些服务由服务提供者提供,供他人使用以交换由服务提供商定义的费用。 所描述的技术有助于以应用提供商和服务提供商配置的方式由应用程序使用这种可调用服务,包括代表最终用户跟踪应用程序对第​​三方可调用服务的使用以及分配收取终端用户的费用 在由应用程序和服务的提供者配置的应用程序和服务之间。 在某些情况下,服务的配置定价条款指定终端用户的费用,这些费用以一种或多种方式与由该服务提供商收取的定义费用不同。

    Data stream ingestion and persistence techniques

    公开(公告)号:US10795905B2

    公开(公告)日:2020-10-06

    申请号:US15858480

    申请日:2017-12-29

    Abstract: A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.

Patent Agency Ranking