A method is disclosed for evaluating a classifier used to determine a traffic light signal state in images. The method includes, by a computer vision system of a vehicle, receiving at least one image of a traffic signal device of an imminent intersection. The traffic signal device includes a traffic signal face including one or more traffic signal elements. The method includes classifying, by a traffic light classifier (TLC), a classification state of the traffic signal face using labeled images correlated to the received at least one image. The classification state controls an operation of the vehicle at the intersection. The method includes evaluating a performance of the classifying of the classification state generated by the TLC. The evaluation is a label-free performance evaluation based on unlabeled images. The method includes training the TLC based on the evaluated performance.