The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system to be provided for supporting higher data rates Beyond 4th-Generation (4G) communication system such as Long Term Evolution (LTE). Provided is an operation method of a base station in a wireless communication system. The method comprises: receiving, from a terminal, at least one piece of information among channel quality information on a resource region allocated to the terminal and non-Gaussian information on a nulling region corresponding to the resource region; and determining a modulation order for the terminal, a code rate, a ratio of the resource region to the nulling region based on the channel quality information and the non-Gaussian information.