A method of decoding an enhanced video stream composed of base layer video access units and enhancement layer video access units, each access unit comprising a plurality of syntax structures, includes passing the syntax structures of the base layer access units to a base layer buffer, passing syntax structures of the enhancement layer access units to an enhancement layer buffer, outputting the syntax structures passed to the base layer buffer in a predetermined sequence, outputting the syntax structures passed to the enhancement layer buffer in a predetermined sequence, and recombining the sequences of syntax structures output by the base layer buffer and the enhancement layer buffer respectively to form a complete enhanced access unit, composed of base layer syntax structures and enhancement layer syntax structures in a predetermined sequence.