Endurance Test: Protection of Corporate Solutions in Focus

Nearly 21,000 test cases per product

A vast amount of time and effort go into testing in the lab. After all, each product is required to undergo the real-world test and the reference test multiple times.

Real-world test: The term "real-world" refers to the malware samples used in the test. On the day of the test, they are collected, sorted and then partially deployed. For this, AV-TEST uses its own honeypots, e.g. unprotected PCs that are attacked while surfing the Web. The team registers the visited and infected websites in a special database and evaluates the type of attack. Likewise, the testers use email honeypots, i.e. multiple mail accounts containing messages with malicious files, links to files or links to websites with malware. Each discovered malware sample is analyzed, i.e. precisely documented in terms of what it does to a system. The resulting action list of a malware sample is used in the test to validate whether the security solution really blocks all actions.

All discovered malware samples are required to be new and not already registered in the databases. If they are already on file, but not more than two weeks old, then they are placed onto the list of possible candidates for the second test phase with the reference set.

The actual real-world test proceeds as follows: The test product is installed on a Windows test PC (AV-TEST uses many totally identical PCs). For consumer software, the lab uses the default configuration of the product. For business products, the lab uses the configuration recommended by the manufacturer. The products can be updated at any time via online update and use their cloud services. A solution can thus rely on help from the cloud if the local detection techniques are exhausted. This mirrors a normal real-life scenario. Afterwards, the security solution is tested with the defined group of zero-day malware. In doing so, records are kept on how the security solution responds.

AV-TEST reference set: For the special set, the experts collect the malware samples according to the same system as with the real-world test. However, the malware samples, usually numbering more than 5,000, are already up to two weeks old and therefore already quite well known. The purpose of the exercise is to evaluate whether the products can perhaps later detect threats, i.e. with a slight delay, that first went undetected.

The reference set naturally does not consist only of one special group, such as Trojans. The lab ensures a good mixed variety with backdoors, bots, viruses, Trojans, worms, downloaders, droppers or password stealers. But the notoriety of a malware threat also plays a role: only if it is also relevant and widely distributed (confirmed by at least two independent sources) is it potentially included in the set. The lab enlists the help of telemetry data and information from companies, research institutions and manufacturers of security software to evaluate whether a malware sample is widely distributed.

In this test as well, the security solution is reinstalled on a PC and activated. Afterwards, the solution scans and detects the attackers (on-demand scan). Any malware not detected in the process is then launched individually afterwards in order to test the on-execution protection. If a product works without static protection, i.e. only with behavior-based detection, then naturally all files are launched in the test. As the malicious files are known for up to two weeks, the products ought to detect them practically without exception. And many products are capable of doing so, but not all. In this test phase as well, each security product always has access to its extended security techniques in the cloud.

Success means successfully blocking malware

In the tests, all individual results per product are registered and evaluated. Only the following scenarios, however, are considered "successful detection":

- Access to the URL is blocked
- An exploit on the website is detected and blocked
- A download of malicious components is blocked
- The launching of malicious components is blocked

In the test, merely detecting or warning against a malicious file or link is not considered a sufficiently successful defense. The access or execution must be effectively blocked.

Important to know: All malware samples, infected websites or emails used in the test were collected by the lab. The laboratory does not rely on material from other sources, i.e. from manufacturers or from other public databases. In order for the test to be the same for all products, the test steps run simultaneously for all products on a series of identical Windows PCs.

The results in detail

In the current endurance test from July to October 2017, the five solutions from Kaspersky and Symantec with two versions each and Bitdefender demonstrated the best performance. In all test phases, they recognized a total of 400 attackers in the real-world test and the 20,590 malware samples from the reference set.

The products from McAfee and Trend Micro each failed to detect only one individual attacker in the real-world test. The solution from F-Secure also had minor detection difficulties with five attackers. In terms of their protection, all the products mentioned achieved the maximum score of six points – the laboratory awards points in order to make it easier to compare and assess the performance of the security products. 

The following solutions from G Data and Seqrite had problems detecting 21 and 22 malware threats respectively, of which already 4 of the attackers each were 0-Day malware.

The free protection module from Microsoft, the System Center Endpoint Protection, failed to detect 51 attackers. For Sophos and Avast, the number of undetected malware samples already reached 107 and 190. The product from Palo Alto Networks was unable to block 287 attacks.