Usability

The usability of the protection products is also examined alongside the protective effect and repair performance tests that focus on threats. Due to the fact that it is difficult to objectively quantify the usability of a product itself, this process particularly focuses on the influence on the usability of the system.

Performance Testing

Functions of the operating system, the protection program and other programs that may be disruptive are closed down prior to the beginning of the test. This includes both automatic updates and planned actions such as scans or backups. The tests are carried out on a limited number of computers that are identical in construction and have been verified as having the same system speeds in order to give all products the same chances.

Every individual action is carried out at least seven times so that a reliable average can be generated. If the standard deviation of the individual values exceeds a specific threshold, this indicates an error and the test is repeated in full. If the test achieves reliable values for all products and all test cases, these are compared with the reference system values and the difference is calculated. This difference then specifies the slowing down of the system in the case of the actions tested.

False Positive Testing

The latest versions of widespread programs such as Mozilla Firefox, Adobe Reader and Flash, Java Runtime Environment, VLC Media Player and other similar programs are used to measure the distraction of the user as a result of too many or false warning messages. The products are regularly changed so that no manufacturer is able to prepare for the list. The test cases are then downloaded from the original website, installed and used. During this process, a log is kept as to whether the protection program displays false warning messages or asks the user if certain actions are permitted or not. Furthermore, the AV-TEST analysis system Sunshine is used to examine whether the program was fully installed and all functions are available. If this is not the case or if certain actions are blocked by the protection software, this is also documented and factored into the assessment. The result shows the number of programs for which warning messages were displayed and how many of these programs were (fully or partially) blocked.

Given the complexity of these tests, the number of test cases used is kept very small. A further test on false detections is therefore carried out in a simplified form. This involves examining around 600,000 to 750,000 test cases in order to achieve a statistically relevant range.

AV-TEST’s internal Flare Archive, which boasts a volume of nearly 10 million files, forms the basis for these tests. This archive includes installation programs and various versions of installed files from a diverse range of current and older programs. Thousands of new programs are added every day, meaning that the archive is extended by more than 100,000 new files every week.

All files are analysed in order to ensure that they really are harmless and can be used for false positive tests. The archive is used to produce a random test amount from the files added over the last few weeks. An on-demand scan is then carried out on these files with every protection program in order to examine how many of the safe files are falsely identified as malware. In this process, files from the grey area (for example remote administration software, password recovery programs or commercial keyloggers) are excluded from the result in order to only count clear false positives.