A method is presented for artificial neural network training. The method includes storing weight values in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix, defining the weight matrix to have an output dimension that is smaller than the input dimension such that the weight matrix has a rectangular configuration, and converting the weight matrix from a rectangular configuration to a more square-shaped configuration by repeating or concatenating the rectangular configuration of the weight matrix to increase a signal strength of a backward pass signal by copying an input of repeated weight elements during a forward cycle pass, summing output computations from the repeated weight elements, updating each of the repeated weight elements according to a backpropagated error or alternatively updating only one of the repeated weight elements by setting all forward values except one to zero during an update pass.