Disclosed are a DLP security system and an operating method thereof. An operating method of a data loss prevention (DLP) apparatus, comprising: converting, into packets, Ethernet signals received from a fail over device that are transmitted and received between an external network and internal network; analyzing the packets to classify the packets into first packets required to be precisely judged and second packets not required to be precisely judged; distributing and allocating a judgment job about the first packet to at least one in-line instance according to a predetermined reference; and allocating the judgment job distributed to the in-line instance in which a fail occurs to the in-line instance which is normally operated when it is verified whether there is an in-line instance which is normally operated in the case where the fail occurs in the at least one in-line instance.