A system and method for automated polyp detection in optical colonoscopy images is provided. In one embodiment, the system and method for polyp detection is based on an observation that image appearance around polyp boundaries differs from that of other boundaries in colonoscopy images. To reduce vulnerability against misleading objects, the image processing method localizes polyps by detecting polyp boundaries, while filtering out irrelevant boundaries, with a generative-discriminative model. To filter out irrelevant boundaries, a boundary removal mechanism is provided that captures changes in image appearance across polyp boundaries. Thus, in this embodiment the boundary removal mechanism is minimally affected by texture visibility limitations. In addition, a vote accumulation scheme is applied that enables polyp localization from fragmented edge segmentation maps without identification of whole polyp boundaries.