A shift-by-wire transmission system includes a body provided with a sensor therein; an operation portion assembly in which a magnet is provided within a detecting range of the sensor, wherein a gear shifting operation portion is rotated about an axis of the body so that the magnet is rotated with the gear shifting operation portion, and the gear shifting operation portion is moved in an axial direction of the body so that the magnet is linearly moved together with the gear shifting operation portion; and a controller configured to detect a variation in magnetic force of the magnet detected by the sensor according to a rotational displacement and a movement displacement of the magnet and detect a gear shifting signal corresponding to a gear shifting operation of the gear shifting operation portion.