Example computer-implemented methods, media, and systems for serving machine learning (ML) models using an asynchronous input/output (I/O) mechanism are disclosed. One example method includes receiving a first request for running a ML model to provide a first prediction. A first green thread is generated responsive to the first request and executed on an operating system (OS) thread to send a first asynchronous remote procedure call (RPC) to a multiple producer single consumer (MPSC) channel. A second request for running the ML model to provide a second prediction is received. A second green thread is generated responsive to the second request and executed on the OS thread to send a second RPC to the MPSC channel. The first and the second asynchronous RPCs are scheduled using a first and a second blocking threads respectively, which are used by the ML model to generate the first prediction and the second prediction.