A gesture recognition system includes an image pick-up device, a processor, an operation engine, an optimal template selection means, and a display terminal. The image pick-up device is for capturing an image containing a natural gesture. The processor is for finding out a skin edge of a skin part from the image, and then classifying the skin edge into multiple edge parts at different angles. The operation engine has multiple parallel operation units and multiple gesture template libraries of different angle classes. These parallel operation units respectively find out gesture templates most resembling the edge parts in the gesture template libraries of different angle classes. The optimal template selection means selects an optimal gesture template from the resembling gesture templates found out by the parallel operation units. The display terminal is for displaying an image of the optimal gesture template. Thereby, marker-less and real-time gesture recognition is achieved.