PyInstaller executable bundling: Attackers use PyInstaller as a means to bundle dangerous Python scripts together with a small Python runtime environment. The EXE file can be executed directly under Windows. Which means that the user does not need to have Python or any libraries installed, nor is it necessary to install anything for the launch. It acts as a particularly effective and sophisticated means of distributing malware elements, since the executable files that are created with PyInstaller are typically used by developers to distribute legitimate software. It enables the dangerous applications to pass unnoticed, because users and security tools will categorize them as something harmless.

In our examples, we use PyInstaller to bundle the Python applications into packages, which then load the dangerous shellcode or a manipulated DDL file later on. The final file is executed on the target system in our test, where it launches the embedded Python interpreter and injects our malware code. In this manner, we can simulate the behavior of ransomware attacks and infostealers.

In the current evaluation, 11 products for consumer users from the following vendors were put to the test: Avast, AVG, Avira, Bitdefender, ESET, K7 Computing, Kaspersky, McAfee, Microsoft, Norton and Trend Micro. Added to this were 9 solutions for corporate users from Acronis, Avast, Kaspersky (with two versions), Microworld, Norton, Qualys, Sophos and Trellix.

Each product was subjected to 5 attack scenarios using ransomware and infostealers. The test experts then awarded the points. This means up to 3 points for fending off ransomware, and up to 4 points for defending against infostealers. Half points are also awarded if a product is capable of mounting a partial defense against an attack. At the end of the test, the highest value in the protection score is 35 points. 

The 10 test scenarios

All attack scenarios are documented according to the standard of the MITRE ATT&CK database. The individual sub-techniques are listed in the MITRE database for “Techniques”, for example, “T1566.001” under “Phishing: Spearphishing Attachment”. Each test step is thus defined among the experts and can be logically understood. In addition, all attack techniques are explained, along with how and why the malware infection occurs and impacts systems. 

11 consumer user packages in the ATP test

The security packages for consumer users of Windows 10 systems needed to fend off a number of attacks in the test. 8 of the 11 products tested were neither tricked by the malware nor the latest attack tactics. The following solutions detected the attackers in all 10 scenarios and deleted the malware: Avast, AVG, Bitdefender, K7 Computing, Kaspersky, McAfee, Microsoft and Norton. 

In the end, the Windows package from ESET also detected all attackers; however, it ran into trouble in four cases. The package managed to detect two malware samples, yet it couldn’t completely stop them. Only through the use of secondary protection modules was it possible to stop the attackers. This resulted in half a point being deducted twice. The scenarios started in a similar fashion for two infostealers. Detection was a success; however, there were problems when it came to stopping the infostealers. The malware was prevented from being installed, but it was not possible to block further malware processes. And in both cases, the attackers eventually prevailed. Since data was stolen, it led to a deduction of 2.5 points in both instances. Altogether, 6 points were deducted from the protection score, which means that the ESET product earned a score of only 29 points.

The problems that Avira and Trend Micro encountered can be explained quickly. Both solutions could not detect one ransomware and one infostealer attack, which meant that the malware was able to inflict damage unhindered. The interesting factor: both products ran into problems in scenarios No. 5 and No. 10. And the point deduction was also the same: each package lost 7 points and achieved a score of only 28 points in the protection category.

A solution needs to achieve the required 75 percent (at least 26.5 points) out of 35 points for the protection score in order to pass certification testing. All 11 packages were successful in achieving this. They also need to regularly participate in the bimonthly Windows tests. Again, all products fulfilled this requirement and were awarded the AV-TEST “Advanced Certified” certificate.

9 corporate solutions put to the ATP test

There were 9 solutions tested for use in the corporate environment. 6 of the 9 products scored full points in the ATP test. In all 10 scenarios, they identified the attackers without fail and earned 35 points each in the protection category: Acronis, Avast, Kaspersky (with two versions), Microworld and Trellix.

The solutions from Norton, Qualys and Sophos were able to detect all dangerous malware; however, they were not able to immediately block the attackers in two scenarios each. It took the extra effort of secondary protection mechanisms to isolate and delete the attackers’ malware. Interestingly enough, all solutions initially had problems handling attack scenarios No. 5 and No. 10. Some of the protection packages for consumer users also encountered issues with these scenarios. This is extremely important for the vendors to know.

The same criteria are applied to the products for corporate users. These solutions need to achieve the required 75 percent (at least 26.5 points) out of 35 points for the protection score in order to pass certification testing. In addition, they need to regularly participate in the bimonthly Windows tests. Since all of the solutions for corporate users fulfilled the requirements, they received “Advanced Approved Endpoint Protection” certification.

Crucial takeaways for several of the test participants

The attack techniques used in the ATP test are different from test to test, but they are always based on current attack campaigns or they are recent tactics observed by experts. The majority of the vendors earn the full set of points in the category of protection thanks to the excellent, and oftentimes flawless, work they put into their respective product. 

In the current test for consumer users of Windows computers, 8 out of 11 products tested make the grade: Avast, AVG, Bitdefender, K7 Computing, Kaspersky, McAfee, Microsoft and Norton. 

And a further 6 out of 9 of the protection solutions for corporate users of Windows systems earned the flawless score of 35 points in the certification testing: Acronis, Avast, Kaspersky (with two versions), Microworld and Trellix.

All other products tested had some difficulties with a few scenarios, which interestingly enough were the same ones in many cases. These are crucial takeaways for the developer teams of these solutions, in an effort to improve on the product in future releases. It also stands to note that, when a solution did not immediately detect the attacker, the secondary protection modules were effectively triggered and eventually protected the Windows systems from any damage. The attackers were only truly successful in 4 out of 200 scenarios tested in the lab, eventually encrypting the systems or stealing data.