ATP: live test against data theft and encryption malware

In the table, each solution is assigned a protection score – a maximum of 35 points is possible. For each ransomware sample detected and stopped completely, up to 3 points are awarded, and for each info stealer, up to 4 points. With 5 variants per attack type, half point scores can also be awarded in the total of 10 scenarios.

The laboratory describes the sequence of defensive actions in a matrix according to the MITRE ATT&CK standard. If a product does not immediately detect the attacker, further defensive actions can take effect and thus stop the attack. The test shows exactly at which step this occurs – or doesn't occur.

The latest attack techniques used in the test

In the recent March-April ATP test, a special attack technique was used, which all malware samples in the 10 scenarios in the lab exploited:

MSBuild (Microsoft Build Engine): The trusted Windows tool is part of the .NET Framework and Visual Studio. It is used to create (compile) executable applications from project files (.proj, .csproj etc.). 

Attackers, however, inject malicious code into a project file and offer it for compilation by the user. During this process, the malicious code is executed as fileless malware in the memory and can thus bypass protective measures, as MSBuild is considered a trusted application.

The 10 test scenarios

All attack scenarios are documented according to the standard of the MITRE ATT&CK database. The individual sub-techniques are listed in the MITRE database for “Techniques”, for example “T1566.001” under “Phishing: Spearphishing Attachment“. Each test step is thus defined among the experts and can be logically understood. In addition, all attack techniques are explained, along with how successful the malware is.

ATP: here's how well consumer user products performed

In the March-April ATP test, there were some negative surprises among the products for consumer users. Of the 8 packages examined, only 3 products provided a seamless defense in all 10 scenarios, thus receiving the maximum 35 points: Kaspersky, McAfee and Microsoft.

Norton recognized 9 attackers, but an info stealer was still able to carry out its evil plan. This led to a 3.5 point deduction. The 10th malware, a ransomware, went unnoticed – another 3 points lost. Norton still achieved 28.5 points for the protection score.

It was a grueling test for the packages from Avast, AVG, Avira and F-Secure. All packages recognized only 8 out of 10 attackers. The products allowed one info stealer and one ransomware sample to pass through without resistance. Interestingly, the malware samples are the same in all cases. Overall, these products lost 7 points and ended up with a protection score of 28 out of 35 points.

All tested products earned the "Advanced Certified" certificate from AV-TEST, as they achieved 75 percent of the maximum 35 points (26.5 points).

ATP: here's how well the corporate user products performed

Although the solutions for corporate users performed much better in the test than products for consumer users, there were some issues. The 5 products from Acronis, Kaspersky (both versions), Microsoft and Microworld completed the test without errors and scored the full 35 points for their protection score.

WithSecure's endpoint package also logged all 10 attackers, but had problems afterwards. The already identified info stealer was only stopped in further steps. This cost a one-point deduction, with a final score of 3 out of 4 points. There was also a problem with one ransomware and the loss of further points. Here as well, the attacker was recognized, but not stopped immediately. WithSecure's product only succeeded in subsequent steps. This resulted in 33 out of 35 points for WithSecure's protection score.

The Trellix endpoint solution detected 9 out of 10 attackers. While an info stealer went about its work undetected, a ransomware was discovered but not stopped. In the end, the test system was encrypted. All in all, Trellix scored 28.5 out of 35 points.

Avast's business solution fended off 8 out of 10 attackers without any limitations. In one instance involving an info stealer and ransomware, the solution had to concede, and the attackers commandeered the test systems undisturbed. The result was 28 out of 35 possible points for the protection score.

All the evaluated solutions were able to receive “Advanced Approved Endpoint Protection” certification, as they achieved 75 percent (at least 26.5 points) out of 35 points for the protection score. Only Acronis did not receive the certificate, as this is reserved for products also certified in the regular monthly test and fulfilling the criteria there.

Conclusion: punishing ATP test leaves its mark

As seen in the table, some of the products for consumer users suffered a devastating loss of points. The packages from Kaspersky, McAfee and Microsoft proved, however, that it is still possible to achieve flawless security. They completed the test with top scores with 10 recognized and fended off attackers, attaining 35 points on the protection score.

The result for company products was significantly better, yet far from perfect. Acronis, the two Kaspersky versions, Microsoft and Microworld worked completely error-free, thus earning the maximum 35 points for the protection score.

Although some products detected the attackers in the respective scenario, they only prevented the disastrous work of the malware in subsequent steps. Conversely, sometimes the defense failed, even though the malware was already identified.

Many of the products tested nonetheless proved that the teams of developers working in security companies are doing a good job and are still a decisive step ahead of the attackers.