A method includes determining one or more potential merges corresponding to a color set Ai and a color set Aj of N color sets, represented by A1 to AN, used in coloring polygons of a layout of an integrated circuit. N is a positive integer, i and j are integers from 1 to N, and i≠j. One or more potential cuts corresponding to the color set Ai and the second color set Aj are determined. An index Aij is determined according to the one or more potential merges and the one or more potential cuts. A plurality of parameters F related to the index Aij is obtained based on various values of indices fi and fj. A parameter F is selected among the plurality of parameters F based on a definition of the index Aij.