Test Modules under Windows

Usability

The focus in the test of security programs is clearly on determining the effectiveness of protection. However, in use for both a home and corporate environment, usability of security products plays a role that should not be underestimated. Because false alarms can disturb the work routine just as much as malware attacks can.

Stage 1: False alarms or blocking when visiting websites

False alarms during work on the Internet, as well as unnecessary blocking of websites are detected in this stage of the usability test. In doing so, the test system targets a selection of several hundred popular websites. It automatically detects whether and when a security program falsely issues warning messages or even blocks access due to a false alarm.

Stage 2: False detections of legitimate software as malware during a system scan

This part of the false alarm test detects the quantity of false detections of harmless files by security programs. In hundreds of thousands of test cases, the products are required to achieve a rate of false alarms as low as possible.

Files for the test originate from a constantly growing 15-year-old archive of known benign software. Files from gray areas (e.g. remote administration software, password recovery programs or commercial keyloggers) are removed from the test set in order to detect only unambiguous false positives.

The test set is generally subjected to an on-demand scan. For all programs without an on-demand scan function, the files are launched in sequence on the test system.

Stage 3 – False alarm test for standard software: false warnings concerning certain actions and blocking of these actions carried out while installing and using legitimate software

In this module, the testers determine the number of false alarms caused by the products when deploying the latest versions of widespread software. This includes the use of popular programs such as Adobe Reader, Flash, Google Chrome, Java, Thunderbird or the VLC Media Player.

In addition to monitoring error-free downloads, the test also looks for smooth installation and problem-free use. Thus, among other things, the testers detect false warning messages, as well as queries from the security products, which require user interaction, disturbing the work routine. As a result of these tests, a complete report is assembled on the number and type of falsely issued warning messages and blocked programs.