A testing technique tests and compares malware detection capabilities of network security devices, such as those commercially available from a variety of cyber-security vendors. Testing is conducted on test samples in a “blind” fashion, where the security devices do not know beforehand whether the test samples are “live” malware or benign network traffic. The test samples are received from a remote server and potentially represent malicious attacks against a testing network. Notably, for truly blind testing, embodiments of the testing technique employ a mixture of malware and benign test samples, as well as addressing subterfuge, to prevent the security devices from being able to reliably determine maliciousness of the test samples based on a source of any of the samples.