Endurance Test: Do security packages constantly generate false alarms?

Friend or foe detection

A security solution is required to offer perfect friend or foe detection. That is why the AV-TEST laboratory spent 14 months – from January 2015 to February 2016 – testing how well security solutions handle this issue. In total, 19 packages for private users and 14 solutions for corporate users were tested.

The testers specified four test items for the "Usability" test, which is "lab speak" as formulated by the engineers in their logs:

- False warnings or blockages when visiting websites
- False detections of legitimate software as malware during a system scan
- False warnings concerning certain actions carried out while installing and using legitimate software
- False blockages of certain actions carried out while installing and using legitimate software

Here is how the test worked

In all test categories, only benign websites, clean files or innocuous known applications were evaluated. After all, the existing number of clean files and applications in the IT world is more clearly defined than those infected. That's why all solutions work with so-called whitelist databases, in which all clean and benign files are stored with fingerprints and hash values. If a program does not immediately recognize a file, it queries the cloud whether that file is already registered. If that is not the case, the file is tagged and categorized as good or bad.

In order to obtain a truly relevant test result in terms of false positives, it is necessary to have a large number of benign test data. The lab fully met this standard:

For each individual solution, over a period of 14 months, some 7,000 websites were visited and 7.7 million files put to the test. In addition, 280 applications were launched twice and it was noted whether a false alarm was indicated, or whether the application was even blocked.

The set of 7.7 million test files naturally contained all new files from popular programs, such as Windows 7 to 10 and Office. If security software classified a new system file from Windows as false, that would have fatal repercussions. That is why the latest versions of these important files are always included in the tests.

Consumer user software: some work perfectly

Despite the high standards in the test, involving large volumes of data, some applications manage to get through without triggering a single false alarm. Avira AntiVirus Pro and Kaspersky Internet Security performed error-free.

This was followed by applications with fewer than 10 false positives in the endurance test, Intel Security, Bitdefender, AVG and Microsoft. But even the security packages at the bottom of the list did not deliver any dramatic results in the endurance test; on the contrary.

The results in detail:

- "False warnings or blockages when visiting websites" did not occur at all throughout the entire test when visiting 7,000 websites.

- "False detections of legitimate software as malware during a system scan". The worst score from Ahnlab V3 Internet Security involved 98 false positive files for 7.7 million test cases. That is a quota of 0.001% – acceptable value, even if there is room for improvement.

- "False warnings concerning certain actions carried out while installing and using legitimate software". 11 out of 19 applications did not trigger any false alarms. 6 solutions indicated only 1 to 3 false warnings in 280 tests. Only the suite from K7 Computing issued a total of 10 false warnings.

- "False blockages of certain actions carried out while installing and using legitimate software". Even here, most of the 19 programs did a good job. While 7 solutions did not falsely block anything, 11 of the security packages did so for 1 to 6 innocuous applications. The Comodo Internet Security Premium blocked 29 times.

Enterprise software: only Kaspersky did an error-free job

Also on the test of 14 security solutions for corporate users in terms of usability, nearly all the products indicated that they hardly make any errors in the friend or foe detection. Only the two Kaspersky solutions Endpoint Security and Small Office Security remained error-free in the tests. However, they only took part in 6 out of 7 test rounds.

Throughout a period of 14 months, the solutions from Sophos, Intel Security and Bitdefender showed an overall error rate of under 10. Yet all other products as well only had a low rate of false positives compared to the volume of test data.

The results in detail:

- "False warnings or blockages when visiting websites" did not occur in the entire test of enterprise security software when visiting 7,000 websites.

- "False detections of legitimate software as malware during a system scan". The worst score throughout all tests in this category came from F-Secure with only 49 false positive files for 7.7 million test files. Good, but not excellent like the Sophos solution, which only falsely detected 2 files.

- "False warnings concerning certain actions carried out while installing and using legitimate software". Only 4 out of 14 programs issued 1 or 2 false warnings on the respective 280 tests. A result that ought to please corporate administrators in particular.

- "False blockages of certain actions carried out while installing and using legitimate software". Also in this test category, the results are extremely good. After 280 tests for each program, the findings were as follows: 8 times, everything was error-free, and 6 times, there were only 1 to 5 false blockages. For comparison: on the consumer solutions, the worst result was 29!

Enterprise software solutions generate fewer false alarms

If we put both tests with security software for corporate users and private users side-by-side, we can see that corporate solutions on average generate fewer false alarms. However, it is worth noting that the manufacturers with products for both groups also have the best test results in both cases. This applies to the products from Kaspersky, Bitdefender, Microsoft, Trend Micro, Symantec and F-Secure.

Overall, however, all the products received a high quality rating in terms of usability. While the lab does always subtract a few points for false alarms, strictly speaking, this is critique at what is already a very high level of performance.

Some manufacturers will be taken aback when they see which programs caused the false alarms. A summary is listed in the top 15 and top 30 tables. These include popular applications, such as Notepad++, Yahoo Messenger or WinRAR. These really are not exotic applications, but rather standard software that everyone ought to know. Given the low number of false alarms, the manufacturers should be able to quickly get a handle on these errors in upcoming program versions.