A method, apparatus and computer program product are provided for learning to generate maps from raw geospatial observations from sensors traveling within an environment. Methods may include: receiving a plurality of sequences of geospatial observations from discrete trajectories; refining a summary entity set representation through iteration over discrete trajectories; determining a drive offset for each of the discrete trajectories based on a comparison of the summary entity set representation to a drive entity set for each of the discrete trajectories; aligning the discrete trajectories to generate aligned geospatial observations based, at least in part, on the drive offset for a respective discrete trajectory; concatenating the aligned geospatial observations; processing the concatenated, aligned geospatial observations using at least one Set Transformer; and generating, from the at least one Set Transformer, map geometries including objects from the geospatial observations.