Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sequential data. In one aspect, a computer-implemented method includes receiving a request to generate a system output for an input data sequence, the input data sequence including a plurality of tokens. One or more tokens may be designated as tokens to be skipped. When a token has not been designated as a token to be skipped, the token is processed using a recurrent neural network to update a current internal state of the recurrent neural network. The system output is generated from the final internal state of the recurrent neural network.