Disclosed herein are system, method, and computer-readable medium embodiments for providing the ability to automate the process of generating load tests used for benchmarking APIs. Rather than having to manually generate load tests for a web service API, a test developer can interact with a test service through a web browser and provide the service an API specification and testing parameters. The test service can analyze the API specification, automatically identified endpoints of the API, and generate load tests according to the expected input/output structures of the endpoints. The automatic load test generation can proceed by referring to a library of test instructions and extracting the portions of the test instructions that work for the identified endpoints.