Abstract:
Virtual resource provisioning may be enhanced by coloring virtual resource instances and/or underlying implementation resources. Particular resource colors may be associated with particular treatments during allocation of implementation resources to virtual resources. There may be different types of colors corresponding to different types of allocation treatment. Exclusory colors may be utilized to reduce clustering of virtual resources with respect to implementation resources. Assignment of exclusory colors to virtual resources can help strike a balance between lower costs through efficient implementation resource utilization and higher fault tolerance through spreading across an available implementation resource pool. Inclusive colors may be utilized to require and/or prefer allocation of virtual resources to implementation resources painted with the inclusive color. Proximity colors may be utilized to enhance a computational performance of a set of virtual resources. Proximity colors may be associated with proximity specifications that define proximity in implementation resource networks.
Abstract:
A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.
Abstract:
A local de-duplication table for at least a particular partition of a data stream is instantiated at a particular ingestion node of a multi-tenant stream management service. A submission request indicating a data record of the partition is received at the ingestion node. In response to a determination that (a) the submission request was received within a de-duplication time window corresponding to the particular partition, and (b) the local de-duplication table does not indicate that the data record is a duplicate, a write operation to store the data record at one or more storage locations of the stream management system is initiated.
Abstract:
Methods, systems, and computer-readable media for orchestration of computations using a remote repository are disclosed. A representation of one or more inputs to a computation is stored in a repository. The computation is assigned to one or more hosts of a plurality of hosts. A representation of program code executable to perform the computation is stored in the repository. A local copy of the one or more inputs is stored on the one or more hosts. The computation is initiated on the one or more hosts using the program code and the local copy of the one or more inputs. The computation is initiated for a plurality of keys. The computation succeeds for one or more keys after the computation has failed for one or more other keys. A representation of one or more outputs of the computation is stored in the repository.
Abstract:
Distributed system resources may be managed by applying user created policies to the resources. To ensure that valid policies are applied, remote validation for the policies may be implemented. A validation event for a policy may be detected. A remote validation agent may be identified for the policy and a validation request sent to the remote validation agent that includes information for validating the policy. The remote validation agent may return a validation result for the policy. If valid, a policy action that triggered the remote validation event for the policy may be allowed. If invalid, the policy action that triggered the remote validation event for the policy may be denied.
Abstract:
Resource data objects describing resources in a system may be maintained in multiple different hierarchies for applying policies to manage the resources. Lookup requests may access the different hierarchies to determine which policies are applicable to a given resource based on the policies identified in each of the hierarchies. Modifications to hierarchies may be performed in isolation so that the application of policies in other hierarchies is unchanged by modifications to a different hierarchy. Access restrictions may be enforced with respect to hierarchies so that different users may be permitted access to different hierarchies for system resource management.
Abstract:
A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.
Abstract:
Multi-party updates may be performed for distributed systems. An agreement request may be received that proposes updates to a distributed system. An authorization scheme for the agreement request may be determined and approvers for the proposed updates identified according to the authorization scheme. Notifications may be provided to the approvers indicating the proposed updates to the distributed system. Responses from the approvers may be evaluated to determine whether the authorization scheme is satisfied for the proposed updates. If the authorizations scheme is satisfied, then the proposed updates may be performed to the distributed system.
Abstract:
A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.
Abstract:
A distributed data store may maintain versioned hierarchical data structures. Different versions of a hierarchical data structure may be maintained consistent with a transaction log for the hierarchical data structure. When access requests directed to the hierarchical data structure are received, a version of the hierarchical data structure may be identified for processing an access request. For access requests with snapshot isolation, the identified version alone may be sufficient to consistently process the access request. For access requests with higher isolation requirements, such as serializable isolation, transactions based on the access request may be submitted to the transaction log so that access requests resulting in committed transactions may be allowed, whereas access requests resulting in conflicting transactions may be denied.