An apparatus and method for designing a relay backhaul channel in a wireless communication system are provided. At least one relay node utilized for communication with respective user devices and at least one relay backhaul channel for conducting in-band half-duplex communication with the at least one relay node are identified. The relay backhaul channel may be an FDM channel, a TDM/FDM channel, or a joint R-PDCCH/R-PDSCH channel. The relay channel is used for communicating with the at least one relay node. The supportable ranks of the R-PDSCH channel may depend on the number of resources reserved for demodulation reference signals in the R-PDCCH region.