公开(公告)号：US10095596B1

公开(公告)日：2018-10-09

申请号：US14975064

申请日：2015-12-18

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles ,  Meng Li ,  Andy Kohn

IPC: G06F9/44 ,  G06F11/263 ,  G06F11/273

Abstract: A software testing framework provides functionality for utilizing pre-existing tests to load and performance test a network service. Methods can be tagged with annotations indicating that they are tests, such as integration tests. The methods implementing the integration tests can also be tagged with other types of annotations that can be used to select individual tests for use in testing, such as annotations indicating whether a test is a positive or negative test, annotations specifying dependencies upon other tests, or annotations indicating that a test is a member of a test suite. The annotations can be utilized in conjunction with test selection criteria to select individual integration tests for use in load and performance testing of the network service. The selected integration tests can be deployed to and executed upon load-generating instances to execute the integration tests and generate requests to the network service at high throughput.

公开(公告)号：US09507681B2

公开(公告)日：2016-11-29

申请号：US14656989

申请日：2015-03-13

Applicant: Amazon Technologies, Inc.

Inventor： Ramakrishnan Hariharan Chandrasekharapuram ,  Carlos Alejandro Arguelles

IPC: G06F11/00 ,  G06F11/263 ,  H04L12/26 ,  G06F11/34

CPC classification number: G06F11/263 ,  G06F11/3414 ,  G06F11/3433 ,  G06F2201/81 ,  H04L43/50

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

公开(公告)号：US09203931B1

公开(公告)日：2015-12-01

申请号：US13854786

申请日：2013-04-01

Applicant: Amazon Technologies, Inc.

Inventor： Choi Yong Ngo ,  Mikhail Khasanov ,  Ramakrishnan Hariharan Chandrasekharapuram ,  Vijay Pratap Singh ,  Carlos Alejandro Arguelles

IPC: G06F15/16 ,  H04L29/06

CPC classification number: H04L67/28 ,  H04L67/02

Abstract: Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request, the test client can extract the embedded copy of the received request that the reverse proxy server generated to determine whether it matches the request that a functioning reverse proxy server generates. Based, at least in part on the result of this comparison, the test client can determine whether the reverse proxy server is malfunctioning.

Abstract translation: 公开了用于测试反向代理服务器的系统和相关联的过程。 后端代理服务器测试系统可以从被测试的逆向代理服务器接收请求。 可以响应于来自测试客户端访问后端服务的请求而生成该请求。 在响应所接收的请求时，后端代理服务器测试系统可以包括接收到的请求的副本。 当测试客户端从代理服务器接收到测试客户端请求的响应时，测试客户端可以提取所接收的请求的嵌入式副本，该代理服务器生成以确定其是否与正在运行的逆向代理服务器生成的请求相匹配。 至少部分地基于该比较的结果，测试客户端可以确定逆向代理服务器是否发生故障。

公开(公告)号：US11310165B1

公开(公告)日：2022-04-19

申请号：US14133559

申请日：2013-12-18

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles ,  Ramakrishnan Hariharan Chandrasekharapuram ,  Blair Livingstone Hotchkies ,  Thomas Lowell Keller ,  Choi Yong Ngo ,  Peter Collin Nix

IPC: H04L12/911 ,  H04L47/70

Abstract: A network-based scalable production load test service may be implemented on a provider network including a plurality of computing devices in order to provide load testing for network-based production systems. In some embodiments, the plurality of computing devices is configured to receive a request to capture to a load test data repository items of transaction data for a network-based production service. In some embodiments, the plurality of computing devices is configured to capture to the load test data repository the items of transaction data. The transaction data include input to the network-based production service over a network. In some embodiments, in response to a load test specification received by the scalable production load test service, the plurality of computing devices is configured to dynamically allocate one or more resources to perform a load test of the network-based production service according to the load test specification.

公开(公告)号：US10452522B1

公开(公告)日：2019-10-22

申请号：US14745212

申请日：2015-06-19

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles ,  Xiaozheng Chen ,  Mahmoud Mahmoud

IPC: G06F11/36

Abstract: Embodiments presented herein provide techniques for dynamically generating synthetic test data used to test a web service. In one embodiment, a service description language document defining a web service may include a test interface definition. The test interface definition specifies rules for generating the synthetic data to use when testing API interfaces exposed by the web service, e.g., to generate synthetic data needed carry out load and performance testing. Including rules for generating test data in the service description language document provides a centralized and authoritative source for both building and testing the web service.

公开(公告)号：US10180836B1

公开(公告)日：2019-01-15

申请号：US14834144

申请日：2015-08-24

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles ,  Kevin Lester Quadros ,  Faizal Sultanali Kassamali

IPC: G06F8/33 ,  G06F8/70

Abstract: Technologies are disclosed herein for generating comments in a source code review tool using code analysis tools. A producer module can be executed in order to obtain source code from a source code review tool. One or more source code analysis modules can then be executed in order to analyze the source code. A reporter module can then store the output of the source code analysis modules as comments in the source code review tool for use by a developer of the source code. The producer, reporter, and source code analysis modules can be executed in response to a request from the source code developer to perform a source code review, by a job scheduler, or in another manner. An application programming interface (API) exposed by the source code review tool can be utilized to obtain the source code and to store the comments associated with the source code.

公开(公告)号：US20170244604A1

公开(公告)日：2017-08-24

申请号：US15589898

申请日：2017-05-08

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles

IPC: H04L12/24 ,  G06F11/34

Abstract: Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host.

公开(公告)号：US09639444B2

公开(公告)日：2017-05-02

申请号：US14467690

申请日：2014-08-25

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles

IPC: G06F11/00 ,  G06F11/263 ,  G06F11/34 ,  G06F11/36

CPC classification number: G06F11/2635 ,  G06F11/3495 ,  G06F11/3688

Abstract: One or more computers is configured to run an end-to-end test including at least a plurality of independent tests of multiple stages of an asynchronous multi-stage data processing system. One of the set of independent tests is configured to send a request for test input data from a test data repository service for a particular stage. A converted version of the test input data is obtained. A comparison of the converted version to the output of the particular stage to verify operation of the particular stage is obtained. The output of the particular stage is transmitted to the test data repository service. One or more computers is configured to provide the test data repository service. The test data repository service is configured to store in the test data storage the output of the particular stage as test input data for a next stage of the asynchronous multi-stage data processing system.

公开(公告)号：US09514034B1

公开(公告)日：2016-12-06

申请号：US14570943

申请日：2014-12-15

Applicant: Amazon Technologies, Inc.

Inventor： David Robert Griffin ,  Carlos Alejandro Arguelles

IPC: G06F9/44 ,  G06F11/36

CPC classification number: G06F11/3672 ,  G06F11/3688

Abstract: Methods, systems, and computer-readable media for ordered test execution to enable faster feedback are disclosed. A likelihood of failure is estimated for individual tests in a set of tests. Based at least in part on the likelihood of failure, an ordered sequence is determined for the set of tests, such that the tests are ordered in estimated likelihood of failure. The set of tests is initiated in the ordered sequence, such that one or more computer-executable programs are subjected to individual ones of the tests. A failure of one or more of the tests is determined prior to performing one or more remaining tests in the ordered sequence.

Abstract translation: 公开了用于有序测试执行以实现更快反馈的方法，系统和计算机可读介质。 在一组测试中对个别测试估计出故障的可能性。 至少部分地基于故障的可能性，为该组测试确定有序序列，使得测试按估计的故障可能性排序。 该测试集以有序的顺序启动，使得一个或多个计算机可执行程序受到单独的测试。 在按顺序执行一个或多个剩余测试之前确定一个或多个测试的失败。

公开(公告)号：US09459980B1

公开(公告)日：2016-10-04

申请号：US13864905

申请日：2013-04-17

Applicant: Amazon Technologies, Inc.

Inventor： Carlos Alejandro Arguelles

IPC: G06F17/00 ,  G06F11/30 ,  G06Q10/06

CPC classification number: G06F11/30 ,  G06F11/3006 ,  G06F11/3414 ,  G06F11/3442 ,  G06F11/3452 ,  G06F11/3457 ,  G06Q10/06

Abstract: Automated methods of creating a predictive test load, using clustering to create the predictive test load from production data, for testing in a production environment. Client requests to a production system are captured and processed into production request data. The production request data undergoes a clustering analysis to determine cluster definitions. In some embodiments, the production request data is turned into vectors and the vectors undergo the clustering analysis instead of the production data. A specification may be received that specifies modifications to be made to the production data. The production request data may be processed using the cluster definitions and the specified modifications to create a predictive test load. In some embodiments, the predictive test load is played to a production system to simulate a predictive load according to a test plan. The test plan may specify the rate at which the test load is replayed.

Abstract translation: 创建预测测试负载的自动化方法，使用群集从生产数据创建预测测试负载，以便在生产环境中进行测试。 对生产系统的客户端请求将被捕获并处理为生产请求数据。 生产请求数据进行聚类分析以确定集群定义。 在一些实施例中，将生产请求数据变成向量，并且向量经历聚类分析而不是生产数据。 可以接收指定对生产数据进行修改的规范。 可以使用集群定义和指定的修改来处理生产请求数据，以创建预测测试负载。 在一些实施例中，根据测试计划，向生产系统播放预测测试负载以模拟预测负载。 测试计划可以指定测试负载重播的速率。