Techniques are disclosed for the improvement of vector quantization (VQ) codebook generation. The improved codebooks may be used for compression in cloud-based video applications. VQ achieves compression by vectorizing input video streams, matching those vectors to codebook vector entries, and replacing them with indexes of the matched codebook vectors along with residual vectors to represent the difference between the input stream vector and the codebook vector. The combination of index and residual is generally smaller than the input stream vector which they collectively encode, thus providing compression. The improved codebook may be generated from training video streams by grouping together similar types of data (e.g., image data, motion data, control data) from the video stream to generate longer vectors having higher dimensions and greater structure. This improves the ability of VQ to remove redundancy and thus increase compression efficiency. Storage space is thus reduced and video transmission may be faster.