A contrast control device is provided. The device of the invention includes a detecting unit, a judging unit, a backlight control unit, a gain generating unit and a contrast gain processing unit. The detecting unit calculates an average value and detects a maximum value of a current image. The judging unit generates a gain index and a brightness index according to the average value. The backlight control unit generates a backlight control signal according to the brightness index. The gain generating unit generates a first contrast gain value and a soft-clamping gain value according to the gain index, the average value and the maximum value and then selects one from the first contrast gain value and the soft-clamping gain value for output as a second contrast gain value according to an input pixel value of the current image. The contrast gain processing unit generates an output pixel value according to the second contrast gain value and the input pixel value.