Whole-product anti-malware testing

I've been testing anti-virus and other security software for a long time. Initially I ran small tests for computer magazines but over the years this activity has built up into a business. Now I operate a specialised lab that performs security testing for magazines, the public, businesses and security companies.

Over the years the way we test has changed dramatically and today we have an industry-leading methodology. Are there things we could do to improve it? Of course. But today we received confirmation that we are doing something right.

We submitted one of our anti-malware test reports to the Anti-Malware Testing Standards Organisation (AMTSO). This compared our approach to testing with the nine principles of testing that the organisation published on 31/10/2008 in a documented entitled "The Fundamental Principles of Testing."

The principles are:

1. Testing must not endanger the public.
2. Testing must be unbiased.
3. Testing should be reasonably open and transparent.
4. The effectiveness and performance of anti‐malware products must be measured in a balanced way.
5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
6. Testing methodology must be consistent with the testing purpose.
7. The conclusions of a test must be based on the test results.
8. Test results should be statistically valid.
9. Vendors, testers and publishers must have an active contact point for testing related correspondence.

The review analysis committee found that our test complied with every one of those principles.

As time goes on more security vendors have gained in insight into the level of detail our tests cover, including the forensic approach we take to verifying whether or not malware has compromised a system. We've found bugs in beta products, problems with long-established releases and our results have shown time and time again that blocking malicious websites is generally more effective than using traditional anti-virus scanning.

We are not going to stand still, though. There are plenty of things we can add to our tests, including methods that will allow us to cover more products and larger numbers of threats. All of these progressions will be documented in future reports.