Deutsch
English
Français
Español
- Share:
ATP endurance test: 31 security products for 6 months in the advanced Windows 10 test
Security products for consumer users and corporate users are faced with a formidable task. They have to protect systems around the clock and deal with the latest threats. Sometimes the products detect the attackers immediately – but sometimes only in a further defensive step. This is precisely the aspect examined by the Advanced Threat Protection test. In this endurance test, 31 products show how well they not only detect attackers but also how effectively they defend Windows 10 systems. The result is exciting and reveals some unforeseen surprises.
31 security products show their continuous performance in an advanced test
Consumer users, along with those responsible in corporate environments, always ponder the same questions: Did I bet on the right horse with my current security solution? Or which solution should I be using? The experts from AV-TEST provide precise answers to these questions with their tests. For many products, the current endurance test also indicates the continuity of their protection performance. In this, the ATP endurance test – ATP stands for Advanced Threat Protection – revealed additional special characteristics. Many of the products examined participated in 1, 2 or 3 tests, so that the results can be displayed in a comparison. The test finds out whether the products can detect the attackers, and if yes, whether they can also stop them – immediately or in later defensive steps.
ATP test: actual attacks, as they occur daily
But the ATP test has another special characteristic: the lab doesn't just simply copy the attackers onto the Windows systems. On the contrary, the experts allow the malware to unfurl its devious attack techniques. For example, after a spear phishing attack, a dangerous attachment gains access to a Windows system. Once there, it sneaks its way into the ongoing process, which in turn has so many rights that it can launch other Windows tools and can abuse them. Below is a list of the attack techniques deployed by the attackers in the tests. They are partly explicitly used or also in combination in chains of attack.
Reflective code injection: Reflective code injection loads the code into the memory of the process. In this manner, reflective code loading can bypass process-based detection methods by concealing the execution of arbitrary code within a legitimate and harmless process. This type of code injection is therefore also fileless.
Fileless malware: Fileless malware means that a dangerous code does not find its way into the system via a file package. Instead, the malicious code writes itself from the memory directly to the Windows registry, for example. The attacker uses the tools already available in Windows, such as PowerShell.
Bring a scripting interpreter: Attackers utilize script interpreters such as the PowerShell to execute malware scripts. Many security programs accordingly monitor the interpreters, however. A new variant is the attack with software interpreters, such as AutoHotkey (AHK). A script installs the interpreter and then executes an AHK script, which loads ransomware or an info stealer into the Windows system.
In the tests, the LuaJIT Just-in-Time compiler for the Lua programming language is also used. Attackers conceal malicious code in Lua scripts, to later execute them on the computers of victims.
Microsoft Software Installer: MSI (Microsoft Installer) is a Windows installation package format that provides an application to be installed, complete with necessary data and control commands in one package for the consumer user's computer. Bad actors conceal malicious files in an MSI file and specify control steps.
NSIS (Nullsoft Scriptable Install System): With the open-source, script-driven tool, Windows software installation programs can be created. Attackers conceal malware and fake files in the installation packets, which are executed, launching ransomware or info stealers.
Code concealed in a LNK file: In a seemingly harmless shortcut file (.LNK), malware code is hidden, which is extracted via the PowerShell bundled within Windows. The code then loads ransomware and info stealers onto the Windows system and executes them.
The protection packages for consumer users showed a strong performance in the 6-month endurance test
In the continuous protection test under Windows 10, only the Bitdefender Ultra version achieved the maximum point score
31 products in the ATP test
All products have participated in 1, 2 or 3 ATP tests. Each individual test is always carried out within 2 months. In this, some products were examined for a full 6 months from January to June 2024 under Windows 10, other solutions for 4 months, and some in a test for only 2 months. Whereas the products reveal interesting results with one test, the tests over 4 to 6 months provide a good overview of their protection continuity.
In the ATP test, the individual steps of detection and defense are recorded and described in a matrix according to the MITRE ATT&CK standard. With ransomware, there are three key steps to recognize, with info stealers there are four actions. The lab awards a half or full point for each step that is thwarted or action that is fended off. This means that a product is able to earn 3 points five times for each detected and liquidated ransomware sample, and 4 points five times for the info stealers. Thus, in a test the highest value in the protection score is 35 points. Products that took part in 2 tests were able to reach a protection score of 70 points, and those involved in 3 tests up to 105 points.
ATP endurance test result: 14 consumer user products
There were 14 well-known protection packages in the ATP test overview on the consumer user tests. Some products had a point difference from the maximum score. In cases such as these, the products in the test committed minor errors, thereby losing points.
In the first group with 3 tests were the 4 packages from Microsoft, Bitdefender, Avast and AVG. The unanticipated surprise in the endurance test: The Windows-embedded Defender Antivirus (Consumer) detected all 30 attacks in the test, achieving a protection score of 103.5 out of 105 possible points. This put Defender at the top of the table. Coming in close behind was Bitdefender with 29 out of 30 detected attacks and 99 out of 105 points. The packages from Avast and AVG also detected all 30 attacks and achieved 94 out of 105 possible points across the 3 tests.
The second group with 2 tests consisted of 8 products. These included the manufacturers G Data, McAfee, Microworld, Norton, PC Matic each with 70 out of 70 possible points. Thus, they detected and fended off all 20 attacks in the 2 tests.
This group also included Avira, ESET and F-Secure. While the packages detected all the attackers, they had occasional problems in the subsequent defensive measures. This cost them valuable points. Avira received 69, ESET 65 and F-Secure 57 out of the 70 possible points.
The third group with one test consisted of only 2 products: Kaspersky and Surfshark. Both products achieved perfect results in the test and were awarded the maximum 35 out of 35 points as their protection score.
ATP endurance test result: 17 corporate user solutions
The ATP test series for corporate user solutions was identical to the consumer user products. In the overview with 1, 2 and 3 tests there are 17 security solutions for companies. Here as well, some products had a point score slightly off the maximum score. In such a case, the products in the test committed minor errors, thereby losing points.
In the first group with 3 tests there were 4 products represented. These included 2 different Bitdefender versions, along with Avast and Check Point. They all detected the 30 attacks error-free. The only product in the endurance test achieving 105 out of 105 possible points was the solution Bitdefender Endpoint Security Ultra, providing a very strong protection continuity. Coming in directly behind was the version Endpoint Security from Bitdefender with 100.5 out of 105 points. Avast followed with an even 100 points, and Check Point with 98 points.
The second group with 2 tests of company products consisted of 8 packages. In the evaluation, all solutions each detected 20 attacks on the systems. The products from ESET, HP Security, both versions from Kaspersky, Qualys, Symantec and WithSecure all received the maximum achievable 70 points for their protection score.
Only Microsoft Defender Antivirus Enterprise lost a point in the test, achieving 69 points.
The third group with one test was somewhat more comprehensive for company products. It consisted of 5 solutions. Among them, Microworld, Sophos and Trellix attained the maximum achievable 35 points for their protection score. Cybereason committed errors and thus only reached 30.5 of the 35 points. Seqrite had the same problem, scoring only 29 out of 35 points.
Strong performances in the first ATP endurance test
The normal, continuous ATP tests always provide clear statements on the security solutions for consumer users and corporate users. They show how well the protection packages fend off attackers in real-life scenarios and how they partly also deploy additional security modules to thwart attackers in the subsequent steps.
The compilation of the tests indicates how well and consistently the products protect consumer users or employees in corporate environments. This was especially manifest in the groups with 3 tests over 6 months. Among the products for consumer users, the Windows-embedded Defender Antivirus achieved the surprising result of 103.5 out of 105 points. The security packages from Bitdefender, Avast and AVG followed close behind with 99 and 94 points respectively.
Bitdefender Endpoint Security Ultra stood out of the crowd in the group of solutions for corporate users with 3 tests: it was the only product in the ATP endurance test to achieve 105 out of 105 points. This was immediately followed by the second Bitdefender version without Ultra, with 100.5 points out of a maximum of 105 points. Yet Avast and Check Point with their 100 and 98 points respectively have nothing to be ashamed of.
The middle group with 2 tests did indeed demonstrate a certain protection continuity, but it was naturally not as substantial as the results from 3 tests. Still, most of the products in this group achieved the maximum score of 70 points. It will be interesting to see in the next test results of these products whether they can uphold this constant level.
The ATP evaluations forming the basis of the endurance test included the following:
