After low-temperature finish rolling has been performed on a steel material having a certain chemical composition, cooling is performed at an average cooling rate of 10° C./s or higher to a temperature of 500° C., rapid cooling is further performed in a temperature range from a Ms temperature to a temperature of (Ms temperature - 200° C.), coiling is thereafter performed in a low temperature range of 250° C. or lower, and the coiled steel sheet is uncoiled and further subjected to rolling with a certain amount or more of rolling load per unit width and the like. Consequently, it is possible to obtain a high-strength hot-rolled steel sheet having a microstructure including, in terms of area fraction, 95% or more of a martensite phase at a position located at ¼ of the thickness of the steel sheet, in which an average aspect ratio of prior austenite grains is 3.0 or more.