According to one embodiment, a magnetic memory element includes a stacked body including first and second stacked units stacked with each other. The first stacked unit includes first and second ferromagnetic layers and a first nonmagnetic layer provided therebetween. The second stacked unit includes third and fourth ferromagnetic layers and a second nonmagnetic layer provided therebetween. Magnetization of the second and third ferromagnetic layers are variable. Magnetizations of the first and fourth ferromagnetic layers are fixed in a direction perpendicular to the layer surfaces. A cross-sectional area of the third ferromagnetic layer is smaller than a cross-sectional area of the first stacked unit when cut along a plane perpendicular to the stacking direction.