The usability of the protection products is also examined alongside the protective effect and repair performance tests that focus on threats. Due to the fact that it is difficult to objectively quantify the usability of a product itself, this process particularly focuses on the influence on the usability of the system.
2 different factors are used to measure this influence:
- The distraction of the user as a result of warning messages in the case of unknown programs or due to false positives.
- The false detection of safe programs as malicious software.
False Positive Testing
The latest versions of widespread programs such as Mozilla Firefox, Adobe Reader and Flash, Java Runtime Environment, VLC Media Player and other similar programs are used to measure the distraction of the user as a result of too many or false warning messages. The products are regularly changed so that no manufacturer is able to prepare for the list. The test cases are then downloaded from the original website, installed and used. During this process, a log is kept as to whether the protection program displays false warning messages or asks the user if certain actions are permitted or not. Furthermore, the AV-TEST analysis system Sunshine is used to examine whether the program was fully installed and all functions are available. If this is not the case or if certain actions are blocked by the protection software, this is also documented and factored into the assessment. The result shows the number of programs for which warning messages were displayed and how many of these programs were (fully or partially) blocked.
Given the complexity of these tests, the number of test cases used is kept very small. A further test on false detections is therefore carried out in a simplified form. This involves examining around 600,000 to 750,000 test cases in order to achieve a statistically relevant range.
AV-TEST’s internal Flare Archive, which boasts a volume of nearly 10 million files, forms the basis for these tests. This archive includes installation programs and various versions of installed files from a diverse range of current and older programs. Thousands of new programs are added every day, meaning that the archive is extended by more than 100,000 new files every week.
All files are analysed in order to ensure that they really are harmless and can be used for false positive tests. The archive is used to produce a random test amount from the files added over the last few weeks. An on-demand scan is then carried out on these files with every protection program in order to examine how many of the safe files are falsely identified as malware. In this process, files from the grey area (for example remote administration software, password recovery programs or commercial keyloggers) are excluded from the result in order to only count clear false positives.