A vehicle door structure 10 includes an idling mechanism 23 which invalidates an operation of a handle 15 controlling a locking mechanism 22 in a slide door panel 14 when a swing door panel 16 is opened, the idling mechanism 23 includes an abutting portion 50 which abuts against the vehicle body 5 at a closed position of a swing door panel and a linkage controlling portion 52 which controls a linkage between the handle 15 and the locking mechanism 22, and the linkage controlling portion 52 maintains the linkage between the handle 15 and the locking mechanism 22 when the abutting portion 50 abuts against the vehicle body 5 and breaks the linkage between the handle 15 and the locking mechanism 22 when the abutting portion 50 separates from the vehicle body 5.