A peer-to-peer special purpose processor architecture and method is described. Embodiments include a plurality of special purpose processors coupled to a central processing unit via a host bridge bus, a direct bus directly coupling each of the plurality of special purpose processors to at least one other of the plurality of special purpose processors and a memory controller coupled to the plurality of special purpose processors, wherein the at least one memory controller determines whether to transmit data via the host bus or the direct bus, and whether to receive data via the host bus or the direct bus.