Protection Solutions vs. Ransomware in the Advanced Threat Protection Test
All major ransomware attacks follow a certain attack pattern of APT groups. That's why the lab at AV-TEST simulated attacks on 14 protection solutions for corporate users with 10 sophisticated Advanced Persistent Threat scenarios, trying to encrypt the systems per ransomware. In doing so, each defensive step was evaluated according to the pattern of the MITRE ATT&CK Matrix. The test revealed: Many corporate products are good system defenders.
Every day, protection solutions for corporate users normally have to detect, classify and fend off thousands of attackers. The lab at AV-TEST already assesses this classical protection every two months with its certification tests for corporate protection software and thus evaluates the consistent security of the products.
In the latest Advanced Threat Protection test, the lab evaluated 14 endpoint protection platforms (EPP) in terms of their performance against new, still unknown attack scenarios with ransomware, as they are normally carried out by hacker groups. Unfortunately, many such attacks have been successful in the past, as evidenced by numerous media reports.
Various security companies have already designated numbers from 1 to 41 for several APT groups, as they also show up under several names. Others stick to a group description. APT38, for example, deployed Hermes ransomware, with AES 256-bit encryption. APT41 bought the ransomware-as-a-service (RaaS) encryptor RaaS and used it to extort companies. The FIN7 group mounted specialized attacks on ESXi virtualization servers of VMware with Darkside ransomware. Also notorious is the TA505 group, which encrypted large volumes of data with the LockerGoga ransomware. This software was used against the Norwegian aluminum manufacturer Norsk Hydro, for example. Many smaller APT groups without special names recurringly rely on variants of successful ransomware, such as REvil, Ryuk, Maze, Conti or NotPetya.
The MITRE website offers in-depth information on APT groups and malware. MITRE maintains a list of all APT groups and offers per "T" number a description of the individual attacks and methods.
In all the scenarios evaluated by AV-TEST, the attack begins via e-mail and a dangerous attachment, which ultimately launches ransomware – which is also the modus operandi of most APT groups. In this test there are 10 different samples of ransomware, which in case of final “Impact” proceed to encrypt the attacked Windows system, and then make their ransom demand. But in these monitored live attacks, many corporate solutions don't allow things to progress that far in the first place. In order for a product to receive the certificate "Advanced Approved Endpoint Protection", it has to earn at last 75% of the protection score points in the test. In this test, it means scoring a minimum of 25.5 points. All 14 products in this test received the certificate.
14 solutions in the Advanced Threat Protection test
An APT (advanced persistent threat) attack normally follows a pattern and is carried out in various steps. If an attack occurs, it may not be noticed in the initial phase, but will be detected based on behavior during subsequent operations. But it is also possible that an attack will be detected, yet not completely prevented. In order to document this, for analysis purposes the lab uses an evaluation matrix modeled after MITRE ATT&CK. This has the advantage that the test method is internationally comparable. The matrix follows the sequence of an attack and, in doing so, enables an evaluation of the key sub steps. In this case, while an attack scenario may involve 6 to 8 steps, only 3 or 4 are evaluated. Because the lab documents various intermediate steps for a better overview, but they are not relevant to actual threat prevention.
The following article also features how the lab evaluates the attacks and what is behind the MITRE ATT&CK Matrix: "New Lines of Defense:EPPs and EDRs Put to the Test Against APT and Ransomware Attacks".
Included in the test are the corporate solutions from Avast, Bitdefender (two solutions), Check Point, Comodo, G DATA, Malwarebytes, McAfee, Microsoft, Seqrite, Sophos, Symantec, VIPRE Security and VMware. Each of the named solutions receives the certification "Advanced Approved Endpoint Protection".
Each product is required to solve 10 test cases, for which up to 34 points can be earned. Achieving the maximum point score for 10 detected attacks were these 8 solutions: Bitdefender (Version Ultra), Comodo, G DATA, Malwarebytes, Microsoft, Sophos, VIPRE Security and VMware.
While Avast, McAfee and Symantec also detected all 10 attacks, they did experience a few minor defense-related problems in later steps. Overall, they earned 32.5 out of 34 points. The solutions Bitdefender (Endpoint Security) and Seqrite showed similar results: scoring 31.5 points each.
Check Point failed to detect one of the attacks, losing the possible 3 points for that category. The product was tested with the anti-virus module only, and without the anti-ransomware capabilities that are designed to address the missed attack. All other 9 attacks, on the other hand, were fended off without any sequence errors: thus earning 31 points.
Sequence and evaluation of 10 attacks
The sequence of an attack depends on the scenario. In this test, the lab deploys 10 scenarios for a ransomware attack, whereby each one is always initiated via an e-mail containing an attachment.
The 10 attacks naturally all tend to vary. Often embedded in the attachment are macros, which in turn run PowerShell scripts. In the scenarios, macros are explicitly enabled, as protection is to be provided not from admin settings but from the EPP products. Moreover, the attachments contain autoexec .NET applications or compiled PowerShell or Visual Basic scripts. These are partly obfuscated as another file type. But one aspect is always identical: In the end, there is the planned launch of the ransomware.
As a general rule, an attack according to the MITRE ATT&CK matrix starts with the "Initial Access" or "Execution" step. Because the attack is either launched automatically by access, e.g. via a macro, or by a user operation. In any case, the protection program has to already react. If it does, it receives one point for the protection score.
Depending upon the type of attack already carried out or even detected, the "Persistence" step of attack occurs in some cases. In this scenario, the malware adds keys in the registry or entries in autostart, for example. Detection of the operation is essential, for instance, in order to clean a system. One point can be earned towards the protection score here as well.
Important: the following "Defense Evasion, Discovery and Collection" steps are only documented in the test, as these generally only involve operations not considered dangerous as such. This includes, for example, the reading in of the directory and folder structure or the collection and copying of files that are not very system-relevant. These operations could also originate from other innocuous applications. The lab records these operations for complete documentation in the MITRE ATT&CK Matrix.
Remote system control
If a system has been hijacked by a script or macro, and subsequently by ransomware, this is followed by the step where the controlling server is contacted. In the sequence, this is the "Command and Control" step of the attack. In this step, the ransomware tries to exploit mostly internal Windows ports in order to establish a channel. This can then be used to receive data, such as additional attack tools, to send data, or to receive new instructions.
The final attack step is always "Impact". This is where the decision is made as to what will happen to system data. If ransomware is launched, at a minimum the most important user data is quickly encrypted or even extracted. The interesting factor: Even if a protection package or solution has recognized the attack steps beforehand, this does not necessarily mean that the attack has been thwarted. Given even half a chance to wreak havoc, most of the time the ransomware damage is severe.
The test shows that in some cases, the infected attachment is recognized and – almost – completely blocked from executing. The consequence is that the attack is successful in small parts, usually encrypting individual files, but not the entire system. In these cases, the full protection point score is not awarded. The chart of the full evaluation shows the products where this was the case. 13 of the 14 products there were able to detect 10 out of 10 attacks, but could not fully block or prevent them. Products capable of this feat received the maximum achievable 34 out of 34 points; others only 32.5 or 31.5 points for 10 detected attacks.
The individual evaluation charts of the products precisely illustrate the attack steps successfully fended off. The individual faults, where applicable, are also listed there.
Conclusion of the Advanced Threat Protection test
The scenarios used in the test correspond to the attack methods with ransomware deployed by APT groups. The evaluation shows that the majority of the products examined are in fact capable of dealing with these attacks. A few manufacturers, however, still need to make tweaks and improvements to their solutions. Because an attack that has only been partially identified can cause damage, leaving behind a compromised system harboring additional risks.
While the test scenarios described above only had to do with ransomware, the lab is preparing additional Advanced Threat Protection tests involving other threats, e.g. script attacks or fileless attacks. In which case the threat may be different, but the test procedure with 10 attack scenarios and an evaluation according to the MITRE ATT&CK matrix will be the same. For those tests, the products will once again have the opportunity to earn “Advanced Approved Endpoint Protection” recognition.
Bitdefender Ultra 1/2
Bitdefender Ultra 2/2
Check Point 1/2
Check Point 2/2
G DATA 1/2
G DATA 2/2
VIPRE Security 1/2
VIPRE Security 2/2