An apparatus and method for determining an optimum modulation scheme and code rate in a communication system using a variable transmission scheme. In a transmitter included in the communication system, an encoder encodes an encoder packet (EP) and a plurality of modulators having different modulation orders modulate sub-packets being codewords output from the encoder. A selector selects one of the modulators by comparing an MPR (Modulation order Product code Rate) being a ratio of an EP size to a number of payload modulation symbols, and comparing the MPR with a threshold.