A ray tracing unit is implemented in a graphics rendering system. The ray tracing unit comprises: processing logic configured to perform ray tracing operations on rays, a dedicated ray memory coupled to the processing logic and configured to store ray data for rays to be processed by the processing logic, an interface to a memory system, and control logic configured to manage allocation of ray data to either the dedicated ray memory or the memory system. Core ray data for rays to be processed by the processing logic is stored in the dedicated ray memory, and at least some non-core ray data for the rays is stored in the memory system. This allows core ray data for many rays to be stored in the dedicated ray memory without the size of the dedicated ray memory becoming too wasteful when the ray tracing unit is not in use.