LiDAR based memory segmentation includes obtaining a LiDAR point cloud that includes LiDAR points from a LiDAR sensor, voxelizing the LiDAR points to obtain LiDAR voxels, and encoding the LiDAR voxels to obtain encoded voxels. A LiDAR voxel memory is revised using the encoded voxels to obtain revised LiDAR voxel memory, decoding the revised LiDAR voxel memory to obtain decoded LiDAR voxel memory features. The LiDAR points are segmented using the decoded LiDAR voxel memory features to generate a segmented LiDAR point cloud.