A method is provided for categorizing a problem report in a repository of a problem reporting system as an enhancement request or as a defect. Based on the problem report being the defect and having a high severity level, determining a score indicating a probability that a text portion of the problem report includes problem reproduction steps. In response to the score indicating the probability of problem reproduction steps, flagging the problem report to automate. For each problem report to automate, copying the text portion of the problem report to an automated test script, whereby the test script runs in a continuous pipeline.