Methods and apparatus for predicting an association between input data in a first modality and data in a second modality using a statistical model trained to represent interactions between data having a plurality of modalities including the first modality and the second modality, the statistical model comprising a plurality of encoders and decoders, each of which is trained to process data for one of the plurality of modalities, and a joint-modality representation coupling the plurality of encoders and decoders. The method comprises selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders, and processing the input data with the joint-modality representation and the selected encoder/decoder pair or pair of encoders to predict the association between the input data and the data in the second modality.