Technologies for providing efficient access to pooled accelerator devices include an accelerator sled. The accelerator sled includes an accelerator device and a controller connected to the accelerator device. The controller is to provide, to a compute sled, accelerator abstraction data. The accelerator abstraction data represents the accelerator device as one or more logical devices, each logical device having one or more memory regions accessible by the compute sled, and defines an access mode usable to access each corresponding memory region. The controller is further to receive, from the compute sled, a request to perform an operation on an identified memory region of the accelerator device with a corresponding access mode. Additionally, the controller is to convert the request from a first format to a second format that is different from the second format and is usable by the accelerator device to perform the operation. Additionally, the controller is to perform, in response to the request, the operation on the identified memory region of the accelerator device with the corresponding access mode. Other embodiments are also described and claimed.