While constructing a timeline for Dennis Technology Labs, the security testing venture I started in 2008, I realised that I've been involved in anti-malware testing for over a decade.
This is because, as a journalist, I used to test anti-malware software every year or so, probably from around 2000.
It seems that some of the rather 'edgy' testing techniques I used in the early days are coming into fashion now that 'Advanced Persistent Threats' (APTs) are perceived as being a significant threat.
Looking through my files I found one of the first anti-virus tests I ever published, in the now defunct Computer Buyer magazine.
The report (PDF), which appeared in 2004, contains this description of how I conducted the tests:
What the article doesn't mention there, but alludes to in the individual product reviews, is that I ran all files and monitored their behaviour to ensure that they were actually a threat to the system."Anti-virus programs should be able to detect the current, high-threat files being sent around the internet. Our test files included some of the most virulent and commonly found viruses, as well as well-known backdoor Trojans and harmful Visual Basic scripts that we generated using well-known virus-generation tools. None of these files should pose any problem to a decent scanner.We compressed copies of each into Zip files, too.Finally, just to add a bit of a challenge, we also used a few tricks to disguise the backdoor files. These tricks rely on freely available executable packers and a wrapper program that can attach the backdoor to another, more innocent file – in our case,Windows’ Minesweeper game.Whoever runs the game can play it, but will also unwittingly hand over control of their PC to the attacker.All the tests were run in Windows XP Professional."
For example, I would try to connect to the BackOrifice Trojan used (but not named) in this test, and control the victim system remotely.
Here's an extract from the Norton Antivirus 2004 product review:
"Even when we increased the heuristic level to the Highest setting, our backdoor Trojans were able to enter and operate on the system unhindered."Clearly such a small test provides the reviewer with time to play with configurations. This doesn't really scale when handling hundreds of malicious URLs and dozens of anti-malware programs.
I received a lot of angst from the anti-virus industry whenever one of these reviews was published. The people I spoke to at those companies really didn't like the idea of testers 'creating' threats, whether or not they used easily-available tools and well-known techniques.
Judging by some of the conversations I'm having with companies producing anti-APT solutions it might be time to start digging out the old BackOrifice, eLiTeWrap and UPX packer tools.