In an intelligent video surveillance system, video processing software performs a number of operations on video data received from a camera, including foreground extraction, shadow removal and object tracking. The foreground extraction stage classifies each pixel of a received frame as representing either foreground or background. Since shadow regions can be wrongly classified as foreground, a two-branch shadow removal operation is employed, comprising weak shadow removal (32) and strong shadow removal (33). The purpose of weak shadow removal (32) is to remove only the most severe shadows in each frame. Conversely, the purpose of strong shadow removal (33) is to remove substantially every shadow present in each frame. By comparing the overlap of foreground regions in the two resulting images, it is possible to identify foreground regions that have fragmented due to strong shadow removal, and thereafter classify them as representing a common foreground object. Foreground objects are better defined in terms of their shape and subsequent processing stages will not treat fragmented regions as representing separate objects.