A mobile device, a controlling method thereof, an image display apparatus, and a controlling method thereof are provided. The mobile device includes: a user interface (UI) configured to receive a user command to control the image display apparatus; a communication interface configured to provide a service according to the user command through an access point (AP) to which the image display apparatus is connected, and perform a direct connection operation with the image display apparatus in response to the service being provided; and a controller configured to control the communication interface to hand the service over to a direct channel connected to the image display apparatus in response to the direct connection operation being completed.