In one aspect, the present disclosure implements a method of ranking images in real-time as the images are being received. In this regard, the method comprises receiving a first and a second images from end users. Then, the first and second images are made available to two or more human annotators from network accessible computing devices. The method provided by the present disclosure then receives designations from each of the two or more human annotators regarding whether the first or second image is preferred. From the received input, a determination is made, in the aggregate, whether the two or more human annotators preferred the first or second image. If the two or more human annotators preferred the first image, the method allocates a rank to the first image that is higher than the second image. On the other hand, if the two or more human annotators preferred the second image, the method allocates a rank to the second image that is higher than the first image.