A system and method are disclosed for automatically classifying images of pages of a source, such as a book, into classifications such as front cover, copyright page, table of contents, text, index, etc. In one embodiment, three phases are provided in the classification process. During a first phase of the classification process, a first classifier may be used to determine a preliminary classification of a page image based on single-page criteria. During a second phase of the classification process, a second classifier may be used to determine a final classification for the page image based on multiple-page and/or global criteria. During an optional third phase of classification, a verifier may be used to verify the final classification of the page image based on verification criteria. If automatic classification fails, the page image may be passed on to a human operator for manual classification.