Certified insane: Anti-malware certifications

[Nearly ten years ago I wrote an article for Computer Shopper magazine about anti-virus (aka anti-malware) certifications. The words, along with my website, have long since shuffled off into archive.org. Here's a copy, which I think stands up pretty well even today, despite the progress that testers (including myself) have made since then.]

Effectively testing APT defences

There is a need to test products that claim to detect and protect against advanced threats.

When incredible claims are made by security products so should we make strong efforts to challenge and possibly validate these claims.

Over the past year and a half I have discussed testing with a range of vendors, testers and potential customers of such products. These discussions have varied from very positive to extremely defensive and illogical on the part of some vendors.

To put it this way, we have some anti-APT kit in the lab ready for such a test but equipment from one or two important vendors is elusive, to say that least.

Such testing has also been a regular point of debate at the Anti-Malware Testing Standards Organization, of which I am currently the Chair, and so two AMTSO colleagues (Richard Ford, Florida Institute of Technology and Gabor Szappanos, Sophos) joined me to write a paper called Effectively testing APT defences.

Gabor and I presented this at the AVAR 2014 conference in Sydney, on the 12th - 14th November 2014.

The paper examines some of the problems that surround testing such technologies (real and merely perceived) and questions the definition of the term "APT" in a constructive way. It also walks the reader through an example of a targeted attack and notes where certain types of testing would be deficient when using such an attack as a test case.

In the presentation, but not the paper, I also demonstrated how a 'baseline' set of tests, using unmodified Metasploit-based attacks, made mincemeat of some well-known anti-malware products. One enterprise solution stopped four out of 25 threats. We were able to obtain reverse shells in 21 cases and, just as an experiment, migrated into the anti-malware agent's own process.

This basic test demonstrated that a range of tools, tactics and techniques could be used to test different levels of protection from actors ranging from 'zero' (almost no skill/resources) to 'Neo' (effectively unlimited skills/resources).

Not every tester is capable of 'Neo' testing and some may not wish to conduct 'zero' testing, but as long as the report explains what approaches were taken and why it's hard to understand why an anti-APT vendor would object to tests that could be considered "too easy."

While the paper does not provide a single, ultimate methodology for anti-APT testing I believe that the document does outline some valid approaches, none of which are, "do not test!"

Anti-virus keeps dying

"AV is Dead."

Three easy words that make an almost guaranteed route to headlines in the technical press.

But what do they actually mean?

That anti-malware software is useless?

Let's dig down into this thorny issue and separate the facts from the marketing messages.

For some reason anti-virus technology seems to divide experts into barely-rational groups of those who think it is completely useless and those who think it (or rather, their preferred brand) is a panacea.

A journalist asked me today about some recent "AV is Dead" headlines. Here's my response.

--

The claim that “AV is dead” is guaranteed to make headlines in the technical press. This is why the claim is made so often, sometimes by companies that actually have anti-malware solutions within their own products.

I think there are three main issues worth exploring:

1. AV signatures
2. Business vs. consumer requirements
3. Post-event protection

AV signatures

Anti-malware products that use only signatures of known malicious files are very limited. No decent AV product designed for endpoint systems works that way. They all have additional protection layers to support this most basic function.

It would be rather remiss to omit the signature system entirely (you’d risk ignoring well-known malicious files, which seems rather silly), but to rely on it is clearly a bad idea.

That’s what the “AV is dead” line always comes down to. It should really be:

“AV products that rely solely on signatures are relatively useless in isolation.”

The FireEye report is clearly focussing on “signature-based AV”, although that is not made clear initially. It also resurrects the diabolically-misjudged Imperva report, which made some basic errors and so suffered a lot of criticism.

In real-world tests run by DTL and other testing organisations anti-malware products are rarely 100 per cent effective but neither are they usually completely useless. Microsoft Security Essentials often appears to be quite weak and, in our tests, always appears at the bottom of the ratings - yet it still seems to stop more than 50 per cent of threats.

The best products stop in excess of 90 per cent of threats, most of which are really nasty things like ransom-ware. That does not sound like “dead”/obsolete.

Business vs. consumer requirements

When making general statements about the effectiveness of AV, commentators usually focus on the needs and resources of large businesses.

I am sure that Symantec’s Brian Dye will do a marvellous job with his response team, but I doubt he’ll be sending those guys into your house or mine to help with a ransom-ware infection. They will be focussed on very large businesses.

Similarly, see how companies that focus on white-listing handle AV in the media. It’s always “dead” but… what about consumers? Can they handle white-listing products?

There are very few such products available for consumers and these are hard and annoying to use. They may scale well for businesses in which a small team handles white-listing for many thousands of employees, but you as an individual are not going to want to handle the white-listing needs of your extended family, even if it’s a large one.

Have you ever tried even the most basic parental control software? It’s very labour-intensive to use in the real world, where very real people (small, demanding children) provide feedback that one cannot ignore.

So anti-malware-based products are clearly one of the few options available for consumers and, as long as those products are not entirely signature-based, they should do a reasonable job of protecting people.

They will be better than nothing, at least, which does not sound like “dead”/obsolete.

Post-event protection

Currently businesses seem to be facing far greater threats than consumers. They are being attacked relentlessly, if we believe the stories, and so malware is likely to infect a system on the network at some point. It may then spread, one way or another, through that network and into others.

This is why products from companies like FireEye, Palo Alto Networks and Cisco don’t just try to prevent the initial infection – they have to be able to detect when an infection has occurred and should alert technical staff that something needs to be investigated. At least a few will use signature-based AV as part of that process (in fact I know that some do).

And why not? We have seen a file appear on Fred’s PC and we can take a signature of that and search the other files on the network for other copies.

That makes a lot of sense and does not sound like “dead”/obsolete.

APT testing like it's 2004

Is it time to start testing anti-virus software using methods I last employed ten years ago?

While constructing a timeline for Dennis Technology Labs, the security testing venture I started in 2008, I realised that I've been involved in anti-malware testing for over a decade.

This is because, as a journalist, I used to test anti-malware software every year or so, probably from around 2000.

It seems that some of the rather 'edgy' testing techniques I used in the early days are coming into fashion now that 'Advanced Persistent Threats' (APTs) are perceived as being a significant threat.

Looking through my files I found one of the first anti-virus tests I ever published, in the now defunct Computer Shopper magazine.

The report (PDF), which appeared in 2004, contains this description of how I conducted the tests:

"Anti-virus programs should be able to detect the current, high-threat files being sent around the internet. Our test files included some of the most virulent and commonly found viruses, as well as well-known backdoor Trojans and harmful Visual Basic scripts that we generated using well-known virus-generation tools. None of these files should pose any problem to a decent scanner.We compressed copies of each into Zip files, too.

Finally, just to add a bit of a challenge, we also used a few tricks to disguise the backdoor files. These tricks rely on freely available executable packers and a wrapper program that can attach the backdoor to another, more innocent file – in our case, Windows’ Minesweeper game. Whoever runs the game can play it, but will also unwittingly hand over control of their PC to the attacker. All the tests were run in Windows XP Professional."

What the article doesn't mention there, but alludes to in the individual product reviews, is that I ran all files and monitored their behaviour to ensure that they were actually a threat to the system.

For example, I would try to connect to the BackOrifice Trojan used (but not named) in this test, and control the victim system remotely.

Here's an extract from the Norton Antivirus 2004 product review:

"Even when we increased the heuristic level to the Highest setting, our backdoor Trojans were able to enter and operate on the system unhindered."

Clearly such a small test provides the reviewer with time to play with configurations. This doesn't really scale when handling hundreds of malicious URLs and dozens of anti-malware programs.

I received a lot of angst from the anti-virus industry whenever one of these reviews was published. The people I spoke to at those companies really didn't like the idea of testers 'creating' threats, whether or not they used easily-available tools and well-known techniques.

Judging by some of the conversations I'm having with companies producing anti-APT solutions it might be time to start digging out the old BackOrifice, eLiTeWrap and UPX packer tools.

Update Windows, free from malware?

How effective is the standard security advice, "update Windows"?

This common advice is given in many articles about PC security, but rarely (if ever) do we see any claims as to how much protection we should expect.

We, at Dennis Technology Labs, wanted to find out how well Windows Updates adds to the protection provided by anti-malware software.

The testers in the lab ran a series of anti-malware tests using live, exploit-based web threats. They visited the malicious websites using systems that were running with no Windows Updates applied, with full updates applied and with a variety of popular anti-malware programs.

The end result (PDF) indicates that using Windows Updates protected against around one third of the 100 threats that we used.

Anti-malware software protected against from around two thirds to nearly one hundred per cent of the same threats.

We released this information at the RSA Conference USA 2014 last week and the response was interesting. Some analysts were surprised at how low the 32 per cent figure for 'Windows Updates only' was.

Others expected the protection level to be lower.

Ultimately we found that the best anti-malware products had most of the threats covered from the start, with and without updating Windows, while a couple of products (AVG and MSE) benefited significantly when the latest Windows updates were applied.

Feel free to download the full report and read PCMag.com's coverage from Neil Rubenking.

Four Fs of Anti-Malware Testing

A practical approach to testing endpoint security products

Late last year I wrote a paper and gave a presentation about testing anti-malware software at The First Workshop on Anti-malware Testing Research (WATeR 2013)  in Montreal, Canada.

The paper is published on the IEEE  Xplore website. You can download it from there if you are a member or wish to pay. Alternatively, you can download a pre-published version from here for free.

I'm sure that it will make fascinating reading. However, maybe my presentational skills need polishing as at least one person fell asleep as I spoke. He assured me later that his fatigue was due to the previous night's alcohol intake and not the material in the presentation...

Abstract— This paper presents a practical approach to testing anti-malware products, focusing on the following four areas:

1. Defining the scope of the test
2. Interpreting the test results
3. Methods of data collection
4. Managing the financial costs of a test.

I will also note a number of common mistakes that testers make and explore some of the technical and non-technical challenges that testers face, including attacks on test methodologies by the anti-malware industry and other third-parties.

While the principles discussed apply to many types of anti-malware testing, on different platforms, this paper addresses specific issues relating to testing anti-malware products that run on x86/64 platforms and exposing them to ‘live’ malware threats that actively attack systems on the internet at the same time.

Three anti-virus testing questions/accusations

When we, at Dennis Technology Labs, publish results from our anti-malware tests internet users can be predicted to raise a number of questions, points of opinion and direct accusations.

Here are three of the most common, along with my responses:

1. The test is rigged because I don't believe that Product X would do well but it did. (Or Product Z performed badly but I know that it's good).

The test is not biased in favour of any vendor or vendor's product. Claims to the contrary should be accompanied by evidence.

We deal directly with all vendors involved in our tests and any corruption on our part would, I have no doubt, be discovered and publicised very quickly.

Our reputation is crucial and cheating in tests really makes no sense from a business perspective.

2. Testing on unpatched systems is pointless and produces worthless results.

Anti-malware tests by all well-known testing labs, including Dennis Technology Labs, AV Comparatives, AV Test and NSS Labs focus on testing the actual security software and not other elements. For this reason no tester runs what we call 'security endpoint' tests with the very latest Windows patches deployed.

To use a tired car analogy, if you were to test tyres you would use sub-optimal conditions, such as wet roads and sharp bends. Similarly, providing vulnerable software used by today's malware allows testers to stress the anti-malware software and determine which products are most effective.

In fact, what we see in terms of threats are far more likely to target third-party applications rather than Microsoft Windows components.

In an experimental test that we ran last year, we found that patching Windows with the very latest updates (on a daily basis) had a small effect in preventing the threats, but not enough to make much difference in a test such as we run.

That said, we're not saying that updating Windows is pointless. Far from it - it makes a lot of sense to fix known vulnerabilities.

3. Why doesn't this test include Product Y?

The way that we test is very detailed and labour-intensive. This means we are quite restricted in how many products we test.

Any anti-malware vendor is welcome to engage with us and discuss the inclusion of its products into the test suite.

More than FPs

Testing anti-malware software involves more than investigating how well the products handle threats. They need to handle legitimate software too.

Last week in Montreal, at the October AMTSO meeting, I presented Dennis Technology Labs' approach to running 'FP' tests (as they are generally known in the industry).

A false positive (FP) is when an anti-malware product incorrectly classifies a legitimate program as being malware. For example, if a product claims that a genuine notepad.exe file is a Trojan called W32.Netopad.AYX then that is an FP.

Our view is that FPs are only part of what needs to be tested. Any time that an anti-malware product fails to allow legitimate software to run without annoying or otherwise distracting the user needs to be taken into account.

Similarly, if the legitimate product is somehow limited in what it can do on the system then that is a factor for consideration.

When an anti-malware program does not simply allow legitimate software to run it is not performing optimally. Also, if it classifies legitimate software as anything other than 'safe' or similar then there is a problem.

In such cases we have a Non-Optimal Classification/Action (NOCA). FPs are a subset of NOCAs. FPs are, in fact, just about the worst type of NOCA. Other options are less serious and some are due to bugs in the anti-malware products rather than anything else.

The slides of the presentation are now available online in PDF format.

Rating anti-malware protection

Would you rather your anti-virus program stopped threats from running on your PC, or that it let then run for a while and then removed them?

The answer for most people is, I am sure you will agree, that it's better to keep the malware off the system in the first place rather than give it a chance to cause damage or steal information.

Some people think that rating the effectiveness of anti-malware products is simple. Either it detected the virus or it did not, surely?

If only things were this simple.

Static testing

Admittedly this was at least partially true with traditional, in which a scanner would assess a folder of malicious files. It would either detect the bad file or it would not.

Of course, it might mis-classify a threat but ultimately who cares? As long as it detects that the file is bad, it has succeeded.

Real-world testing

When an anti-malware test uses live threats that attempt to infect the target things change dramatically.

This is because there are different layers of protection involved. While one or more may fail, others can come in to play. But, as we will see, it's better to stop the threat in top layers.

Let's take web threat testing as an example, because that's what we spend a lot of time doing at Dennis Technology Labs. In such tests the target (aka 'victim') computer browses a website that is hosting malicious content. This could be an automatic drive-by attack or some form of social engineering that tricks users into downloading the malware.

When you visit the site while running an anti-malware product a number of things may happen: These include:

1. Product blocks the website, denies access to its content and prevents the attack.
2. Product allows the website but detects and blocks the malicious content.
3. Product allows the website, including some malicious content. However, it successfully detects and removes the main payload before any damage is done.
4. Product allows the website, including all malicious content. As malicious files are executed and others are downloaded the product starts detecting and removing some. Ultimately the system is cleaned, although there may be some inactive files left on the hard disk.
5. Product allows the attack to run fully.

There are, of course, other possibilities but these are the ones that we see quite frequently. In some cases (2 - 3) the product may be so good that it completely removes any traces of the attack. That is possible, but not usual, when the threat has managed to run on the system. It is always in the case when the product blocks the website fully (1).

It's pretty clear that disallowing the threat from running on the system is the best possible case. Any interaction between a threat and its target is dangerous. For example, data may be stolen while the threat runs, even if the malicious files are removed from the system later.

Rating the differences

It is for this reason that we allocate different scores depending on the outcome. We give the highest scores to the products that keep the threats off the system completely, give credit to those that remove the threats completely and acknowledge those that disable the threat, even if some traces remain. Obviously we penalise those products that allow the threat to run.

In our reports we use the following terms as shorthand to describe these cases:

• Defended
• Neutralised
• Compromised

We also use the term, "with full remediation," which means, "no traces of the threat remained on the system."

With a 'defended' result full remediation is implied, as there never were any traces to be found or removed. In some (rare) cases of 'neutralisation' we may determine that all traces were removed.

Here is an extract from one of our 2013 reports that explains things in more detail:

These levels of protection have been recorded using three main terms: defended, neutralized, and compromised. A threat that was unable to gain a foothold on the target was defended against; one that was prevented from continuing its activities was neutralized; while a successful threat was considered to have compromised the target.

A defended incident occurs where no malicious activity is observed with the naked eye or third-party monitoring tools following the initial threat introduction. The snapshot report files are used to verify this happy state.

If a threat is observed to run actively on the system, but not beyond the point where an on-demand scan is run, it is considered to have been neutralized.

Comparing the snapshot reports should show that malicious files were created and Registry entries were made after the introduction. However, as long as the ‘scanned’ snapshot report shows that either the files have been removed or the Registry entries have been deleted, the threat has been neutralized.

The target is compromised if malware is observed to run after the on-demand scan. In some cases a product might request a further scan to complete the removal. We considered secondary scans to be acceptable, but continual scan requests may be ignored after no progress is determined.

An edited ‘hosts’ file or altered system file also counts as a compromise.

The full picture

To fully rate an anti-malware product one also needs to factor in how it handles legitimate software. In the worst cases does it misclassify useful programs as malware? How much responsibility does it pass on to the user? (e.g. unhelpful messages that say "application.exe wants to run. Allow or deny?")

Dennis Technology Labs uses a Total Accuracy rating to
show how effective products are at protecting against
threats and handling legitimate software

To gain a full overview of how effective an anti-malware product is you need to measure not only its detection rates but its protection rates. The different possibilities mean that it is rarely as simple as a 'hit' or a 'miss' and product ratings in tests should reflect that.

What is a malware sample?

Malware samples come in many
forms. This is not one of them.

After we run an anti-malware test some security companies request the malware samples that their products failed to recognise.

I'm never quite sure what they mean when they use the phrase 'sample' because it can mean different things.

Here's a short list of the most common options.

1. Malicious program
2. Hash of a malicious program
3. URL
4. Hash of URL
5. Network capture, aka 'pcap'
6. Web session, aka 'replay'

At Dennis Technology Labs we deal mainly with the sixth option - the web session. In addition to providing web session replays we also make the fifth (pcap) option available in some cases.

As we test all layers of security, from web reputation systems down to file detection scanners, providing samples like this enable security vendors to verify our results and possibly improve their products.


1. Malicious program

Usually such samples are referred to as a binary, an executable (aka an 'exe') or a PE file. In practical terms these types of samples are either a downloader or the payload downloaded by the downloader. You would expect to see files named in a similar way to those below:

0132787483643.exe
foto(4).exe
xyz-britney.scr
winlive.exe


2. Hash of a malicious program

Instead of sending collections of malicious files around the internet, sometimes it's sufficient to simply identify the file using a mathematical hash.

This can be useful because most anti-malware vendors have massive databases containing details of all known (to them) malware and these database records usually include a hash for each file.

You can generate a hash of a file using one of many free utilities, such as MD5sums. To discover the MD5 hash value of a file called test.txt you might type the following command in Windows:

C:\> md5sums test.txt

The output would look something like this:

MD5sums 1.2 freeware for Win9x/ME/NT/2000/XP+
Copyright (C) 2001-2005 Jem Berkes - http://www.pc-tools.net/
Type md5sums -h for help

[Path] / filename  MD5 sum
---------------------------------------------------
[C:\]
test.txt           a0895e00f4d49c355f4f33f69475f963

3. URL

This could be as simple as www.example.com, or a more detailed (and arguably more useful) example could be www.example.com/dir/bad.exe or www.example.com/dir/script.php.

Some testers download files from the web to enable web reputation systems to protect the system. This is a good idea, although using drive-by download and social-engineering pages is more realistic than a direct download taken out of context, such as typing www.example.com/dir/bad.exe into the browser, rather than clicking on it from a malicious email or internet messaging message.

4. Hash of URL

As with files, it is possible to generate hashes of URLs (see '2. Hash of a malicious program' above).

5. Network capture, aka 'pcap'

Saving all of the network traffic generated during an attack makes it possible to recover downloaded files and to understand why a product may not have been working properly. If it fails to send a successful query to its back-end database, for example, the pcap file may contain evidence to help uncover this issue.

Packet capture files, which is where the name 'pcap' comes from, contain as much or as little information as you choose when you start monitoring the network. We capture every packet, so the files are large but contain everything. Capturing just the packet headers is useful for troubleshooting but you can't pull binaries out of the resulting (tiny) files.

6. Web session, aka 'replay'

It's possible to capture a complete web session using a tool like Fiddler. Such capture files include any exploit code used in a web attack as well as programs that are downloaded.

The benefit for security companies is that they can reproduce the same attack as when we used it to test anti-malware products. You can't do that with just the program itself, or a hash of that program.

To replay a Fiddler capture file you can use a utility like Microsoft's HTTPREPLAY.

Anti-virus testing conspiracy theories

Every time an anti-virus tester publishes a report the WildersSecurity internet forum buzzes with a mixture of responses ranging from conspiracy theory to logical reasoning (via understandable suspicion, disappointment and useful questioning).

A recent line of discussion following the latest Dennis Technology Labs report is no exception and includes some pretty direct accusations of corruption, as well as an increasing number of sensible replies.

I have posted my response on the forum, but here is a copy of my statement:

Hello everyone.

As usual I have enjoyed the discussion and would like to provide some feedback.

I appreciate that some people reading this thread have preconceived ideas, or even conspiracy theories, that will never be changed by anything that I write here. However, for the record, I would like to answer some of the questions, concerns and both direct and implied accusations that appear above.

Prior associations with Symantec, “money talks”. Dennis is a Symantec affiliate. Mock surprise that Norton comes top.

We have associations with most vendors through our involvement in AMTSO. We have provided services to companies including, but not limited to, McAfee, Trend Micro, Kaspersky Lab, ESET, AVG and Symantec.

We pride ourselves on our transparency and the ethical way in which we conduct our business. We have no secret affiliations with any customer.

One of the reasons the vendors like working with us is because our testing can provide a number of benefits. For example, if a product performs very well then the vendor’s marketing team are happy. However, if we discover a problem then the engineers are able to use the very detailed information that we log to fix that problem.

It would be of very limited use to everyone if we fixed the results to favour one vendor. Such cheating would be fairly obvious to the other vendors, and the favoured vendors would be unaware of problems with their products.

Assumption that most tests used internet malware

Absolutely, and this fact was made clear throughout the report, starting with the introduction. We expose the products to live (internet) web-based threats. Products that block malicious URLs thoroughly will do well, whereas those that rely solely on signatures and heuristics face a greater chance of being compromised.

DTL tests are not on a par with the major test labs

All test labs have different methodologies. It would be fairly pointless if we all worked in exactly the same way as we would be duplicating each other’s work.

Some testers concentrate on running vast number of samples through AV engines. We focus on investigating cases in forensic detail. This means that our sample numbers are much smaller than those found in on-demand tests. The workload is heavy and manual. It takes us six weeks to run a 100-sample test with 20 or less products.

Having said that, when other testers perform ‘real-world’ testing such as we specialise in, they also use small sample sizes that are comparable with ours.

The note that the test is unsponsored was an attempt to knock Wilder’s argument up front. Also, unsponsored is a matter of definition.

Important though your opinions are to us, we don't word our reports to pre-empt predictable controversies on Wilders. And as you can see, it would not work anyway . We include such details to ensure that everyone knows when a test is and is not sponsored. We try to be as transparent as possible.

Sometimes a customer will ask us to perform a test. This often happens at certain times of the year, such as in the summer when new products are launched. If we agree that the products in the list are fairly comparable we will consider running such tests. These, which are commissioned by the customer, are what we call ‘sponsored’ tests.

The regular quarterly testing is financed by subscribers, which we refer to as partner vendors. These companies support the test by paying a quarterly subscription fee. In return they are allowed certain privileges.

For example, if their product performs well then they are able to use our logos to promote their success. They also have an opportunity to review our logs to discover where problems may lie with their products.

Finally they have an opportunity to challenge our findings (for their own products). This means that, should they wish to argue with our findings, they can do so before we publish the report. This is why you sometimes see comments from a vendor in the report itself – perhaps explaining why they believe the product suffered some issues.

Neutralisation definition is vague: “What constitutes a running threat ? A malicious script that is terminated/blocked as soon as it starts, or some malware that is already active on the computer (e.g. running trojan)?”

A neutralisation occurs when a threat runs on the system but is unable to effect a significant change. As we log every process that runs on the test systems we know if a malicious process ran or not.

A script or Trojan that executes, but which is terminated soon after it starts, sounds like a classic example of a neutralisation in our test.

There is also a further granularity to this type of result. We have the possibility of partial or full remediation.

So if a script ran, was terminated and no changes were made to the system then that would be a neutralisation with full remediation. This means extra points.

However, imagine that a Trojan runs and sets itself up to survive a reboot using a Registry Run entry. The AV product detects the Trojan and removes it, but leaves the Registry entry in place. Or removes the Registry entry but leaves the Trojan file on the disk. That would be a neutralisation without full remediation.

The testing platform is outdated (Win XP) and runs vulnerable software. Hopes that Dennis will switch to Windows 7.

As we noted in the report, there are a lot of people still running Windows XP. According to the research that we’ve seen, just less than half of all Windows users are using XP.

As XP’s life is coming to an end, and Windows 7 has recently become dominant, we will switch to Windows 7 before the end of this year. You can expect one more XP-based report (2013 Q2) before we upgrade.

Regarding the use of vulnerable software, we pre-install it to give the security software a chance to protect the system.

To use an analogy: if we were testing car tyre safety we would not do so in optimum conditions. We'd have windy, wet roads. Similarly we use systems that are vulnerable to attack so we can judge how well the security software protects them.

Who are the partner vendors?

As I mention above, we work with all of the main AV vendors but I am not going to name the specific companies that form the partner vendor line-up.

In some cases you can easily tell, because you’ll see our logos on their marketing material. But in other cases either they may choose not to show the logos or, more likely, they may not achieve an award at all.

Desire for paid AVG.

We have tested the paid version of AVG in two earlier tests. There was a strong desire from readers to see free products and, as the paid-for and free versions of AVG *should* receive similar results it seemed sensible to test the free version only.

Why do I think that the free and paid-for versions should receive similar results? Because we are not currently testing email threats or sending malware and exploits over IM. Ours is a web-based test.

Default settings used?

With consumer products default settings are used. However, we disable each product’s ability to report its statistics and other telemetry home to the vendor. Also, we may increase the logging level if that helps us discover issues with the product.

In some cases we may extend the number of TCP ports that the product is allowed to scan for HTTP-based threats. This is because we use a range of ports when replaying the threat sessions.

More programs should have been tested

This consumer test includes nine very well-known products. At the same time we ran a small business and enterprise test, bringing the total number of products to 19.

This involves a vast amount of work and, while we could possibly squeeze a few more products in, it’s not a trivial matter. We hope that as more vendors support the test we'll be able to extend the number of products tested.

Adjusted testing procedures to give benefit to skew the test in favour of one or more products/vendors
We do not adjust our methodology in favour of any vendor or product. We will make small configuration changes if a vendor notes that its product is behaving differently in our environment than it would in the real world.

Our test setup is very close to being the same as many home and business users, but there are always compromises we have to take in order to expose the products to the same threats in the same way. Malicious sites don’t like being tested!

So when we encounter issues we address them to remain fair to all.

DTL tests are on a par with basic magazine tests and are not comparable with AV test labs’ work

Dennis Technology Labs is a testing business that resides within the media company called Dennis Publishing Ltd. We provide test data to magazines within Dennis and to magazines published by other companies.

We test anti-malware products constantly, using state of the art techniques and equipment. I don’t know of any magazine that would be able to summon resources even vaguely resembling the way we work.

As I mentioned above, we are focussed on realistic testing and ensuring that all results are accurately verified and recorded. We don’t run on-demand scans on millions of files like some testers do. We simply take a different approach.

That said, our results are not that different in terms of rankings, to those produced by the other major testers. This is pretty interesting considering our different testing approaches.

No regulation of testing groups

There is no regulation of anti-malware testing, just as there is no regulation of many industries. There is the Anti-Malware Testing Standards Organization, though. AMTSO brings together vendors and testers into a forum for (usually very robust!) discussion.

Dennis Technology Labs is proud to be a member of AMTSO, alongside almost all of the other major testing groups.

From my own, personal point of view, one of AMTSO’s great strengths is that it enables discussion between vendors, testers and anyone else interested enough to engage. It’s not a regulator but it is very influential.

DTL Anti-Virus Protection tests (2013_Jan-Mar)

Dennis Technology Labs has released its third set of unsponsored anti-virus reports.

This  includes the 2013 versions of popular consumer products.

The tests examine the abilities of anti-malware software for consumers, small businesses and enterprises to protect against infected websites.

The independent tests expose PCs to real internet threats in real-time, using the same infected websites that attack users’ systems across the world. Testers then monitor how effectively different security programs protect the systems.

The results are available to download for free.

Dennis Technology Labs is six

Six years ago today I launched Dennis Technology Labs.

Today we have a great team working on testing dozens of products all year round, using advanced forensics techniques.

It was not always so.

The very first version of the lab was a small box containing two servers, a switch and four Shuttle PCs. It sat in the corner of the editorial office, next to a publisher's office. People would walk past and casually observe world-leading anti-virus products failing to block malware. They were always surprised.

You can marvel at this system's glory above right, or here for slightly more detail.

While the latest version features significantly more equipment and is vastly more scalable, sadly it lacks the exciting blue and green fluorescent lighting.

We still have the original biohazard sign, though.

Introduction to malware forensics

Earlier this month I was delighted to sit on a discussion panel at Kaspersky Lab's reviewers workshop in Lisbon.

I later demonstrated some tools and techniques for analysing the behaviour of malware. I have provided links to the tools and some other useful resources.

(left to right) Peter Stelzhammer, Sveta Miladinov, Simon Edwards and Andreas Marx

The area that I focussed on involved detecting rootkit-related malware in tests. Usually you can't tell if there are hidden processes running using conventional tools.

At Dennis Technology Labs we always check for hidden processes using some advanced techniques usually reserved for malware analysts and forensics investigators.

To get a short insight into how we do that (and how you can), take a look at the link above.

Following a series of interviews, Arne Arnold from PC-Welt wrote an interesting article that shows the difference in opinions of testers and analysts. [Original (German); a poor Google Translation (English)]

Is it infected?

How do you know if a system is infected with malware?

This question is important to journalists, testers and other reviewers of anti-malware software.

The security product may claim to have defeated the threat but you need to dig down into the system using forensic tools to be sure that it has succeeded.

The following links and notes are intended for the journalists who attended Kaspersky Lab's reviewers workshop this week:

Wireshark
http://www.wireshark.org/download.html
http://wiresharkdownloads.riverbed.com/wireshark/win32/Wireshark-win32-1.8.6.exe
http://wiresharkdownloads.riverbed.com/wireshark/win64/Wireshark-win64-1.8.6.exe

CaptureBAT
https://www.honeynet.org/node/315
https://www.honeynet.org/files/CaptureBAT-Setup-2.0.0-5574.exe

>> CaptureBAT.exe -l demo.txt -n -c

Autoruns

http://technet.microsoft.com/en-US/sysinternals


WinPrefetchView
http://www.nirsoft.net/utils/win_prefetch_view.html
http://www.nirsoft.net/utils/winprefetchview.zip

Volatility
https://code.google.com/p/volatility/
https://volatility.googlecode.com/files/volatility-2.2.standalone.exe

>> volatility-2.2.standalone.exe -f stuxnet.raw pslist

>> volatility-2.2.standalone.exe -f stuxnet.raw psscan

>> volatility-2.2.standalone.exe -f stuxnet.raw psxview

Malware Analyst's Handbook
http://www.malwarecookbook.com
http://goo.gl/7gONZ (specific page on Amazon.com)

Stuxnet analysis
http://mnin.blogspot.co.uk/2011/06/examining-stuxnets-footprint-in-memory.html

DTL's test lab in 2013

Although we expanded the test lab just before New Year 2013 we've only now found time to take some photos.

This is because we've been busy ploughing through the first test for 2013, the results for which will be available in the next few weeks.

The main change to the lab is that we used to house the testers and the computers all in one cramped, hot and airless space. Now that room is devoted solely to computers. We have turned on the strong air conditioning, which was too uncomfortable for humans to bear, so the computers are happier too.

A second, larger adjoining room has been created for the testers to use. They are able to control the test computers remotely from this much more comfortable environment. The air conditioning is pleasant and the noise is considerable reduced. Here's what it looks like:

The main testing room can
accommodate at least eight testers

Just a few of our test PCs. There are more
on the other side of the computer room.

Application control test results

For nearly a year we at Dennis Technology Labs ran a test that looked deeply into the benefits and downsides to the whitelisting approach to application control.

I discussed the results, how we tested and some of the issues that we encountered at Kaspersky's Reviewers Roundup in Madrid last week.

Left to right:
Vicente Diaz (Kaspersky moderator),
John Hawes (Virus Bulletin),
Simon Edwards (Dennis Technology Labs)
and Peter Stelzhammer (AV Comparatives)

Kaspersky has just launched a video in which I explain some of the more important elements of the test, and why whitelisting may be coming of age.

DTL Anti-Virus Protection tests (2012 Q4)

Dennis Technology Labs has released its second set of unsponsored anti-virus reports.

These now include the 2013 versions of popular consumer products.

The tests examine the abilities of anti-malware software for consumers, small businesses and enterprises to protect against infected websites.

The independent tests expose PCs to real internet threats in real-time, using the same infected websites that attack users’ systems across the world. Testers then monitor how effectively different security programs protect the systems.

The results are available to download for free.

UPDATE
The following press coverage is available:

Microsoft Security Essentials Tanks Another Antivirus Test
Neil Rubenking (PC Mag)
7th February 2013

Microsoft Security Essentials flunks another antivirus test
Nicole Kobie (PC Pro)
7th February 2013

Dennis virus lab expansion

Dennis Technology Labs has had the busiest year to date so we need to expand.

The lab is now de-constructed ready for expansion into another part of the building. This will increase  physical security while also improving the working environment for both testers and computers.

What have we done this year?

In 2012 we launched our quarterly anti-malware testing service, which has received great media coverage and plaudits from those working in an industry that is very critical of tests.

For over eight months we tested application control solutions using a brand new methodology that is both realistic and very fair. I expect to publish that report soon.

We continue to provide bespoke anti-malware effectiveness and performance testing, while building new tests including an advanced infection remediation test that is able to answer questions like, "how effectively will my anti-virus program fix an infected PC?"

Additionally we have spent a lot of energy testing security deployed in Virtual Desktop Infrastructure environments. This is a challenge because we essentially need to set up multiple enterprise-grade virtual networks. We achieved this prohibitively expensive goal using some pretty special, great-value Supermicro kit supplied by Boston.

What next?

All this work needs a pile of equipment and a group of motivated people who have great attention to detail. With this in mind we have decided to enlarge the lab, putting all of the computers in one room and most of the testers in another, quieter room.

This should make testing a more pleasant experience and, most importantly, the computers will enjoy superior air conditioning!

The lab is now dismantled and awaiting the electricians and builders. The photo below shows just some of the equipment that we've been using throughout the year. We store the testers elsewhere.

New anti-virus test results launched

Dennis Technology Labs has released its first unsponsored anti-virus reports.

These examine the abilities of anti-malware software for consumers, small businesses and enterprises to protect against infected websites.

The independent tests expose PCs to real internet threats in real-time, using the same infected websites that attack users’ systems across the world. Testers then monitor how effectively different security programs protect the systems.

There are already some good news stories reporting on these tests and their results:

Neil Rubenking wrote on his PCMag.com SecurityWatch column,

"London-based Dennis Technology Labs contracts to perform in-depth testing, but doesn't typically make the results public. Today, for the first time, Dennis Labs has released a public report on their testing of eight significant security products; the results are quite interesting."

John Leydon from TheRegister.com says,

"The Dennis Labs results went beyond on-access and on-demand scanning results and probed a wider range of blocking techniques found in modern anti-malware products. Using less usual (arguably more comprehensive) criteria also meant some well-regarded consumer security products that normally test well performed poorly."

David Ludlow, editor of Computer Shopper and Expert Reviews published this,

"DTL – part of the same company as Expert Reviews – performs real-world testing, exposing anti-virus software to threats in the same way as users would be, making its results the most accurate. For this test every single product was exposed to the same 100 live threats."

Darien Graham-Smith from PCPro also wrote about the test reports.