Vapor cells are disclosed that include a dielectric body having a first surface and a second surface. The dielectric body includes a plurality of walls extending from the first surface to the second surface. A perimeter wall surrounds an open volume of the dielectric body and interconnected walls are arranged within the open volume to partition the open volume into a plurality of cells. Each cell has a first opening defined by the first surface and a second opening defined by the second surface. The vapor cells additionally include a first optical window covering the first openings and having a surface bonded to the first surface of the dielectric body to form a seal around each of the first openings. A second optical window covers the second openings and has a surface bonded to the second surface of the dielectric body to form a seal around each of the second openings.