An apparatus and method for data transmission in an Orthogonal Frequency Division Multiplexing (OFDM) communication system are provided. An OFDM system includes a reference Relay Station (RS) and a cooperative relay station respectively for performing an Spatially Layered transmission Mode (SLM) process by modulating a phase and magnitude of a signal by using a predetermined modulation order suitable for the reference relay station to obtain a spatially-layered signal and for delivering the spatially-layered signal to a target Mobile Station (MS); a Base Station (BS) for selecting the reference relay station and the cooperative relay station among a plurality of relay stations; and an mobile station for detecting data through Maximum Likelihood (ML) detection in which a signal having the largest ML is detected by comparing the received spatially-layered signal with a signal which is converted through the SLM upon receiving the spatially-layered signal from the reference relay station and the cooperative relay station.