Certified insane: Anti-malware certifications

[Nearly ten years ago I wrote an article for Computer Shopper magazine about anti-virus (aka anti-malware) certifications. The words, along with my website, have long since shuffled off into archive.org. Here's a copy, which I think stands up pretty well even today, despite the progress that testers (including myself) have made since then.]

How was Hacking Team hacked?

The attacker who stole Hacking Team's data gained access to an employee's computer while the victim was still logged in.

The attacker either had direct physical access to Christian Pozzi's PC or they used malware to achieve a similar level of access. Whichever way it was, we can tell that Christian was logged in at the time simply by looking at a folder name among the files that were leaked onto the internet.

12 computer security tips

The bad guys can try to break in using computers only or they can try to trick you into providing them with access. Or they can use a mixture of tactics. Let's consider two main types of attack:

* Technical – breaking in via computers only
* Human – tricking people into aiding the attack

For the technical attacks you might consider the following, in order of priority/effectiveness:

Vulnerable security software

Last week the Superfish debacle became news and PC manufacturer Lenovo was slammed for pre-installing adware on new laptops.

Since then I've had people ask me about how dangerous this stuff actually is and whether or not security software that works in similar ways poses similar threats.

Pre-infected smartphones

Some Android mobile phones are being sold pre-infected with malware.

According to a blog post by Lookout's Jeremy Linden, "DeathRing is a Chinese Trojan that is pre-installed on a number of smartphones most popular in Asian and African countries."

Some of the most important points from his report include:

• Detections are moderate in volume.
• Detections are global.
• The Trojan pretends to be a ringtone app.
• The Trojan downloads SMS and WAP content.
• The downloaded content can be used to trick users.
• Most of the affected devices are counterfeit or uncommon models (in the West).
• Anti-malware software cannot remove it.

The main lesson to learn here is to buy non-counterfeit devices from reputable sources. You might also consider installing an anti-malware product to alert you to problems. And watch for unexpected charges on your mobile phone bill.

[Image: Sad Android by Justin Marden]

Effectively testing APT defences

There is a need to test products that claim to detect and protect against advanced threats.

When incredible claims are made by security products so should we make strong efforts to challenge and possibly validate these claims.

Over the past year and a half I have discussed testing with a range of vendors, testers and potential customers of such products. These discussions have varied from very positive to extremely defensive and illogical on the part of some vendors.

To put it this way, we have some anti-APT kit in the lab ready for such a test but equipment from one or two important vendors is elusive, to say that least.

Such testing has also been a regular point of debate at the Anti-Malware Testing Standards Organization, of which I am currently the Chair, and so two AMTSO colleagues (Richard Ford, Florida Institute of Technology and Gabor Szappanos, Sophos) joined me to write a paper called Effectively testing APT defences.

Gabor and I presented this at the AVAR 2014 conference in Sydney, on the 12th - 14th November 2014.

The paper examines some of the problems that surround testing such technologies (real and merely perceived) and questions the definition of the term "APT" in a constructive way. It also walks the reader through an example of a targeted attack and notes where certain types of testing would be deficient when using such an attack as a test case.

In the presentation, but not the paper, I also demonstrated how a 'baseline' set of tests, using unmodified Metasploit-based attacks, made mincemeat of some well-known anti-malware products. One enterprise solution stopped four out of 25 threats. We were able to obtain reverse shells in 21 cases and, just as an experiment, migrated into the anti-malware agent's own process.

This basic test demonstrated that a range of tools, tactics and techniques could be used to test different levels of protection from actors ranging from 'zero' (almost no skill/resources) to 'Neo' (effectively unlimited skills/resources).

Not every tester is capable of 'Neo' testing and some may not wish to conduct 'zero' testing, but as long as the report explains what approaches were taken and why it's hard to understand why an anti-APT vendor would object to tests that could be considered "too easy."

While the paper does not provide a single, ultimate methodology for anti-APT testing I believe that the document does outline some valid approaches, none of which are, "do not test!"

Regin: When did protection start?

Regin, advanced malware that is most likely a government espionage tool, is making headlines.

This is because it's a very well-constructed set of tools and also because observers are surprised at how successful it was. It also targeted GSM networks, which is novel.

The big question is, how could the major anti-malware firms have missed this threat for so long?

Or, one might ask, did they really miss it or quietly detect it?

Some people appear to believe that, as Regin was probably created and used by Western governments, then Western anti-malware companies colluded to ignore the threat.

Symantec seems to have been slow to notice Regin because its write-up of Backdoor.Regin claims that it was discovered in December 2013, which is much later than March 2011, when Microsoft updated its definitions to include Regin.A.

In an effort to find a history of Symantec's detection of this malware I obtained an archive of Regin samples from security researcher Claudio Guarnieri and asked the kind folk at VirusTotal to discover when, if ever, Symantec's scanner first detected each sample.

Before we look at these results I want to be clear about what these results mean and what they do not because VirusTotal data is easily abused and dodgy conclusions readily-reached.

The table below indicates that Symantec's technology was capable of detecting most of the samples as being at least suspicious from February 2010. It then made a clearer classification of being a 'Trojan' from March 2011.

Only yesterday (24th November 2014) did it officially label the threat as 'Regin'. This corresponded to its announcement of the Regin threat.

Usually the problem with using VirusTotal is that someone will upload some files, show that product X failed to recognise them and then conclude that the product, or the entire anti-virus industry, is useless.

In this case we can see dates relating to when the product detected the files as threats. Possibly the product would have protected against these files even earlier, and possibly those that appear as having been missed (Classification = 'nothing') would have been stopped through some other layer of protection not related to file signatures.

So I see the following as a worst-case scenario. Symantec's scanner recognised most of these files as threats from around 2011 onwards. Maybe it was capable of stopping them and maybe not - we can't know that for sure. But it's fair to assume that if a signature-based scanner can recognise a file then it will probably generate an alert at the very least.

I've focussed one Symantec simply because it first announced the Regin malware, minutes before other vendors joined in.

Mobile phone kill switches

Microsoft and Google have just announced so-called kill switches for Windows Phone and Android devices.

Apparently this is a response to a reduction of iPhone thefts reported since Apple introduced Activation Lock into iOS 7.

Kill switches are great in principle but hard to implement properly.

Don't assume that a kill switch will 'brick' a phone.

A kill switch is a way to render a lost or stolen phone useless, but currently they don't really do that. Or not fully, at least.

Users can lock or wipe Android devices remotely using third-party apps and, more recently, the Android Device Manager. Windows Phone users will soon see similar abilities included in the Find My Phone feature.

Wiping resets the device and that effectively means the thief (or whoever finds it) gets to keep a fresh and functional gadget. A locked smartphone can be reset too. The data may be wiped but someone else gets the benefit of your expensive hardware.

It's been possible to lock and wipe mobile devices from the major vendors for a while now. The real news is that these security features will soon be enabled by default.

The official statement from Attorney General Schneiderman includes, right at the end, the following:

"Because kill switches are only available on an opt-in basis, not enough consumers are signing-up. This underscores the urgency... to make kill switches a standard opt-out function on all phones."

So we've had kill switches for a while and now they will be opt-out. So far so good.

The kill switches used or planned by Apple, Google and Microsoft are based on software. As we know, there are ways to resurrect a phone locked down by software.

A more effective option would be to build new phones with hardware kill switches. Imagine a fuse that can be blown remotely. The device would then be properly 'killed' and would require expensive repairs to restore the device.

Hardware kill switches bring their own problems.

1. How do you test that they work without destroying your new phone?
2. If you can't test the kill switch, how confident are you that it will work when you need it?
3. You might 'brick' your lost phone and then find it down the back of the sofa.

If introducing software security features reduces thefts of expensive devices then it's worth doing, but users should be aware of the limitations inherent in this approach.

One final thought: if an unauthorised individual or organisation was to gain access to your account they would be able to kill your phone, tablet or other mobile device. If the kill switch was a software version then you'd experience significant inconvenience. If it was of a hardware type you'd face additional cost.

Why don't people encrypt email?

Most PGP/GnuPG reviews/howtos I've seen start with long description of public key cryptography and instructions on key generation.

And that's the problem.

(I've been using PGP/GPG for 20 years and I still use a cheat sheet).

Are Chromebooks insecure travel companions?

Do not connect a Chromebook to a public wireless network if you don't want to risk leaking personal information.

This advice is not some half-baked journalistic opinion but comes direct from the UK government’s National Technical Authority for Information Assurance (CESG).

The CESG advises that even for data with the lowest security classification [PDF], labelled 'OFFICIAL', Chromebooks are unsuitable for a number of reasons, one being that the Virtual Private Network (VPN) included is not up to scratch and there are no viable alternatives.

Yesterday the organisation published its End User Devices Security Guidance: Chrome OS, which states:

"The VPN has not been independently assured to Foundation Grade, and does not currently support some of the mandatory requirements expected from assured VPNs. The VPN can be disabled by the user and some Google traffic is sent prior to the VPN being established resulting in potential for data leakage onto untrusted networks. Without assurance in the VPN there is a risk that data transiting from the device could be compromised"

There is a similar warning in the Android guide that the in-built VPN for Android 4.4 has not been assured, although it does not appear to leak data automatically and there are alternatives available.

If you don't use a VPN when using public WiFi then you are at no greater risk when using a Chromebook than any other device, but if you are security conscious and want to stay safe when moving around at home and abroad it's better to stick with an Android tablet than move over to a Chromebook.

Three Android Security Essentials

I recommend the following three tools for improving your security when using Android phones.

These will allow you to:

1. Locate or disable a lost or stolen phone
2. Prevent surveillance of your WiFi and mobile data use
3. Further improve the security of your internet accounts

Additionally protection from malware is provided by the second tool but, for Android, I don't believe that such threats are currently significant for users who avoid unofficial application stores.

Two tools are free. I've included one that is not because I believe that the free alternatives are not sufficient.

Google Android Device Manager
Why: Ring, lock or erase your phone remotely
Cost: Free
Where: Already installed or Google Play Store

Install the application if necessary and visit the management console using another computer (phone, tablet or PC), logging in with a Google account that is present on your phone.

If you lose your phone nearby you can locate it using its ringer. If you lose it further afield you can locate it using GPS. If it's really lost or stolen you can wipe it.

More information, including a video on what the wiping process looks like.

F-Secure Freedome
Why: Use public WiFi networks safely
Cost: £20.99/year (or £2.99/month)
Where: Google Play Store

Prevent others from monitoring your internet use, passwords and other sensitive data by clicking the Protection button.

When it says 'On' your connection to the internet is encrypted, even if the websites you use do not use encryption.

Additionally this product claims to protect against web-based threats including harmful applications, websites and sites that track your online activity.

If you are considering a free VPN bear in mind that you don't necessarily know who is running it and what information they are taking from you to fund it.

Google Authenticator
Why: Increased security for your internet accounts
Cost: Free
Where: Google Play Store

Not only available for Android, this security tool makes your internet accounts much harder to crack.

Compatible services include Google Apps and Mail; Dropbox; LastPass; and WordPress. That is just a short list of those using Authenticator.

Anti-virus keeps dying

"AV is Dead."

Three easy words that make an almost guaranteed route to headlines in the technical press.

But what do they actually mean?

That anti-malware software is useless?

Let's dig down into this thorny issue and separate the facts from the marketing messages.

For some reason anti-virus technology seems to divide experts into barely-rational groups of those who think it is completely useless and those who think it (or rather, their preferred brand) is a panacea.

A journalist asked me today about some recent "AV is Dead" headlines. Here's my response.

--

The claim that “AV is dead” is guaranteed to make headlines in the technical press. This is why the claim is made so often, sometimes by companies that actually have anti-malware solutions within their own products.

I think there are three main issues worth exploring:

1. AV signatures
2. Business vs. consumer requirements
3. Post-event protection

AV signatures

Anti-malware products that use only signatures of known malicious files are very limited. No decent AV product designed for endpoint systems works that way. They all have additional protection layers to support this most basic function.

It would be rather remiss to omit the signature system entirely (you’d risk ignoring well-known malicious files, which seems rather silly), but to rely on it is clearly a bad idea.

That’s what the “AV is dead” line always comes down to. It should really be:

“AV products that rely solely on signatures are relatively useless in isolation.”

The FireEye report is clearly focussing on “signature-based AV”, although that is not made clear initially. It also resurrects the diabolically-misjudged Imperva report, which made some basic errors and so suffered a lot of criticism.

In real-world tests run by DTL and other testing organisations anti-malware products are rarely 100 per cent effective but neither are they usually completely useless. Microsoft Security Essentials often appears to be quite weak and, in our tests, always appears at the bottom of the ratings - yet it still seems to stop more than 50 per cent of threats.

The best products stop in excess of 90 per cent of threats, most of which are really nasty things like ransom-ware. That does not sound like “dead”/obsolete.

Business vs. consumer requirements

When making general statements about the effectiveness of AV, commentators usually focus on the needs and resources of large businesses.

I am sure that Symantec’s Brian Dye will do a marvellous job with his response team, but I doubt he’ll be sending those guys into your house or mine to help with a ransom-ware infection. They will be focussed on very large businesses.

Similarly, see how companies that focus on white-listing handle AV in the media. It’s always “dead” but… what about consumers? Can they handle white-listing products?

There are very few such products available for consumers and these are hard and annoying to use. They may scale well for businesses in which a small team handles white-listing for many thousands of employees, but you as an individual are not going to want to handle the white-listing needs of your extended family, even if it’s a large one.

Have you ever tried even the most basic parental control software? It’s very labour-intensive to use in the real world, where very real people (small, demanding children) provide feedback that one cannot ignore.

So anti-malware-based products are clearly one of the few options available for consumers and, as long as those products are not entirely signature-based, they should do a reasonable job of protecting people.

They will be better than nothing, at least, which does not sound like “dead”/obsolete.

Post-event protection

Currently businesses seem to be facing far greater threats than consumers. They are being attacked relentlessly, if we believe the stories, and so malware is likely to infect a system on the network at some point. It may then spread, one way or another, through that network and into others.

This is why products from companies like FireEye, Palo Alto Networks and Cisco don’t just try to prevent the initial infection – they have to be able to detect when an infection has occurred and should alert technical staff that something needs to be investigated. At least a few will use signature-based AV as part of that process (in fact I know that some do).

And why not? We have seen a file appear on Fred’s PC and we can take a signature of that and search the other files on the network for other copies.

That makes a lot of sense and does not sound like “dead”/obsolete.

APT testing like it's 2004

Is it time to start testing anti-virus software using methods I last employed ten years ago?

While constructing a timeline for Dennis Technology Labs, the security testing venture I started in 2008, I realised that I've been involved in anti-malware testing for over a decade.

This is because, as a journalist, I used to test anti-malware software every year or so, probably from around 2000.

It seems that some of the rather 'edgy' testing techniques I used in the early days are coming into fashion now that 'Advanced Persistent Threats' (APTs) are perceived as being a significant threat.

Looking through my files I found one of the first anti-virus tests I ever published, in the now defunct Computer Shopper magazine.

The report (PDF), which appeared in 2004, contains this description of how I conducted the tests:

"Anti-virus programs should be able to detect the current, high-threat files being sent around the internet. Our test files included some of the most virulent and commonly found viruses, as well as well-known backdoor Trojans and harmful Visual Basic scripts that we generated using well-known virus-generation tools. None of these files should pose any problem to a decent scanner.We compressed copies of each into Zip files, too.

Finally, just to add a bit of a challenge, we also used a few tricks to disguise the backdoor files. These tricks rely on freely available executable packers and a wrapper program that can attach the backdoor to another, more innocent file – in our case, Windows’ Minesweeper game. Whoever runs the game can play it, but will also unwittingly hand over control of their PC to the attacker. All the tests were run in Windows XP Professional."

What the article doesn't mention there, but alludes to in the individual product reviews, is that I ran all files and monitored their behaviour to ensure that they were actually a threat to the system.

For example, I would try to connect to the BackOrifice Trojan used (but not named) in this test, and control the victim system remotely.

Here's an extract from the Norton Antivirus 2004 product review:

"Even when we increased the heuristic level to the Highest setting, our backdoor Trojans were able to enter and operate on the system unhindered."

Clearly such a small test provides the reviewer with time to play with configurations. This doesn't really scale when handling hundreds of malicious URLs and dozens of anti-malware programs.

I received a lot of angst from the anti-virus industry whenever one of these reviews was published. The people I spoke to at those companies really didn't like the idea of testers 'creating' threats, whether or not they used easily-available tools and well-known techniques.

Judging by some of the conversations I'm having with companies producing anti-APT solutions it might be time to start digging out the old BackOrifice, eLiTeWrap and UPX packer tools.

Retro: Windows 98 end-of-life

I wrote the following column for Computer Shopper magazine when Microsoft ceased providing security updates for Windows 98.

It seems appropriate to republish it now, as Windows XP is on the verge of the same fate.

October 2006

Computer Shopper magazine #224

Rants & Raves

WINDOWS 98 RIP

SIMON EDWARDS RANTS

So long, Windows 98. Rest in peace. When you were born you made getting online easier, you made using large hard disks possible and you were easier to install.

You had a new desktop system that not only allowed users to display web content live from slow and expensive dial-up connections, but regularly caused the system to crash.

You helped destroy Netscape’s business with your integrated web browser, simultaneously heralding the beginning of years of security problems for Windows users, and your integrated Media Player was judged anti-competitive, too.

So long, Windows 98. Rot In Pieces.

After eight years, Microsoft has pulled the plug on Windows 98. No longer will the company provide updates, even if (or rather, when) further security holes are found. The same goes for Windows Me. That’s OK, though, because no-one uses Windows 98/Me any more. No-one except the estimated remaining 70 million users.

98 TEARS

So who is using Windows 98 these days? My guess would be a mixture of people who are non-technical and those who can’t afford to buy a PC every few years. The original date for termination of Windows 98 support was 2003. It is thought that pleas from the developing world persuaded Microsoft to issue updates for a further three years. There will also be plenty of people who could afford a new computer but don’t see any reason to do so – if it still works, why worry?

If these people don’t use the internet, there is no real reason to upgrade. As long as they’re happy with their PC’s performance, don’t share files with anyone and don’t try anything clever with wireless networks then they should be reasonably safe to continue.

For now, you can still buy Windows 98-compatible firewalls and antivirus software, and the free Firefox web browser is also available from www.mozilla.org, so most internet risks can be reduced, even if you do want to browse the web and collect email safely using your old PC.

Radio 5 Live called me this month to find out if listeners should just buy a new Windows XP PC and have done with it. I wanted to say, “No, just keep your thirdparty security software up to date, install a different web browser and email client and it will be OK”. But buying a new PC is the easiest way forward. The fact that you can buy a reasonable Dell laptop for £399 makes the decision easier still.

HASTA LA VISTA

There is an argument that you should wait for Windows Vista to arrive before buying a new PC because, before you know it, Windows XP will be at the end of its life, too. But that means spending hours locking down your Windows 98 machine (and are you really sure you’ve done it right?) for just six months of operation before Vista supposedly arrives. Is it worth your time?

If you are very technically minded, you could install a Linux distribution on your old PC and breathe new life into it. You might even find it runs faster. You might also find you can’t get it working at all, but that’s all part of the fun.

The missus has a laptop that runs Windows 98. It’s a small, slim, silver machine that runs fast and performs the basic internet-based tasks for which it was bought. I am ashamed to admit that it doesn't even have a firewall or anti-virus software installed on it, although it does operate behind a hardware firewall.

It has never shown any symptom of viruses or spyware*, mainly because it is only used for about half an hour a week to check a Gmail account, which is scanned for viruses by Google. We discussed the options, including hardware upgrades and the Linux route.

This discussion was short, and I was made clearly aware that the way forward involved the minimum amount of time with the computer in pieces on the living room floor and the minimum number of late nights featuring me swearing at installation screens in the spare room.

We decided to buy a cheap £399 replacement laptop that will run Windows XP Home for the next few years. If you are one of the 70 million Windows 98 users out there, that is my advice to you, too.

* I am aware of how naive this sounds these days.

Update Windows, free from malware?

How effective is the standard security advice, "update Windows"?

This common advice is given in many articles about PC security, but rarely (if ever) do we see any claims as to how much protection we should expect.

We, at Dennis Technology Labs, wanted to find out how well Windows Updates adds to the protection provided by anti-malware software.

The testers in the lab ran a series of anti-malware tests using live, exploit-based web threats. They visited the malicious websites using systems that were running with no Windows Updates applied, with full updates applied and with a variety of popular anti-malware programs.

The end result (PDF) indicates that using Windows Updates protected against around one third of the 100 threats that we used.

Anti-malware software protected against from around two thirds to nearly one hundred per cent of the same threats.

We released this information at the RSA Conference USA 2014 last week and the response was interesting. Some analysts were surprised at how low the 32 per cent figure for 'Windows Updates only' was.

Others expected the protection level to be lower.

Ultimately we found that the best anti-malware products had most of the threats covered from the start, with and without updating Windows, while a couple of products (AVG and MSE) benefited significantly when the latest Windows updates were applied.

Feel free to download the full report and read PCMag.com's coverage from Neil Rubenking.

Future anti-virus support for Windows XP

Microsoft will no longer support Windows XP after April 2014. In an earlier article I outlined how XP users may secure their systems should they continue to use the obsolete operating system.

I mentioned that some anti-malware programs can help because they may prevent attackers from gaining entry through some of the security holes that are known to exist in Windows.

Many vendors support very old versions of their products. At the very least they provide signatures to customers running ancient scanners. If enough people continue to use Windows XP then those vendors will support them to some extent, even if they don't continue developing new detection and prevention technologies for Windows XP products.

Below is a list of anti-virus companies and the level of their stated commitment to Windows XP users. The following data is for consumer products only.

'Foreseeable future' means 2016 or later.

Vendor	July 2015	Mid-2015 or later	Foreseeable future
AVG			X
Avira			X
BitDefender		X
Bullguard			X
Checkpoint (ZoneAlarm)			X
ESET			X
F-Secure			X
G Data			X
Kaspersky Lab			X
Lavasoft (Ad-Aware)			X
McAfee			X*
Microsoft	X**
Panda Security			X
Symantec (Norton)			X
Trend Micro			X
Webroot	-	-	-

Last updated: 04/02/2014 1325 GMT

* Opinion, based on historical behaviour of this vendor when dealing with end-of-life operating systems.
** This support is available to users who install Microsoft Security Essentials up to April 2014, after which users will be unable to download the software.

Any vendors who are not represented above can contact me for inclusion in this chart. I will also be happy to update the above positions should they change.

The information above is based on publicly-available press material and discussions between myself and the vendors in question.

Four Fs of Anti-Malware Testing

A practical approach to testing endpoint security products

Late last year I wrote a paper and gave a presentation about testing anti-malware software at The First Workshop on Anti-malware Testing Research (WATeR 2013)  in Montreal, Canada.

The paper is published on the IEEE  Xplore website. You can download it from there if you are a member or wish to pay. Alternatively, you can download a pre-published version from here for free.

I'm sure that it will make fascinating reading. However, maybe my presentational skills need polishing as at least one person fell asleep as I spoke. He assured me later that his fatigue was due to the previous night's alcohol intake and not the material in the presentation...

Abstract— This paper presents a practical approach to testing anti-malware products, focusing on the following four areas:

1. Defining the scope of the test
2. Interpreting the test results
3. Methods of data collection
4. Managing the financial costs of a test.

I will also note a number of common mistakes that testers make and explore some of the technical and non-technical challenges that testers face, including attacks on test methodologies by the anti-malware industry and other third-parties.

While the principles discussed apply to many types of anti-malware testing, on different platforms, this paper addresses specific issues relating to testing anti-malware products that run on x86/64 platforms and exposing them to ‘live’ malware threats that actively attack systems on the internet at the same time.

Exploring web threats

How to examine malicious websites and their effects - for professional beginners.

If you want to capture a live sample of a phishing website, or a site that is infected with malware, the techniques covered will help.

These tips are particularly useful considering how malicious websites can come and go very quickly.

I wrote this presentation for a customer who wanted a way to analyse some of the main threats to its users and to help when problems occurred. The solutions and threats included:

1. Phishing websites designed to steal account details.
2. Sites hosting exploits and malware designed for stealing account details.
3. Malware infection detection.
4. Malware removal tips.

You can download a free PDF of the presentation I gave.

If you just want an introduction to HTTPReplay and Fiddler2 then this might be helpful also.

Three anti-virus testing questions/accusations

When we, at Dennis Technology Labs, publish results from our anti-malware tests internet users can be predicted to raise a number of questions, points of opinion and direct accusations.

Here are three of the most common, along with my responses:

1. The test is rigged because I don't believe that Product X would do well but it did. (Or Product Z performed badly but I know that it's good).

The test is not biased in favour of any vendor or vendor's product. Claims to the contrary should be accompanied by evidence.

We deal directly with all vendors involved in our tests and any corruption on our part would, I have no doubt, be discovered and publicised very quickly.

Our reputation is crucial and cheating in tests really makes no sense from a business perspective.

2. Testing on unpatched systems is pointless and produces worthless results.

Anti-malware tests by all well-known testing labs, including Dennis Technology Labs, AV Comparatives, AV Test and NSS Labs focus on testing the actual security software and not other elements. For this reason no tester runs what we call 'security endpoint' tests with the very latest Windows patches deployed.

To use a tired car analogy, if you were to test tyres you would use sub-optimal conditions, such as wet roads and sharp bends. Similarly, providing vulnerable software used by today's malware allows testers to stress the anti-malware software and determine which products are most effective.

In fact, what we see in terms of threats are far more likely to target third-party applications rather than Microsoft Windows components.

In an experimental test that we ran last year, we found that patching Windows with the very latest updates (on a daily basis) had a small effect in preventing the threats, but not enough to make much difference in a test such as we run.

That said, we're not saying that updating Windows is pointless. Far from it - it makes a lot of sense to fix known vulnerabilities.

3. Why doesn't this test include Product Y?

The way that we test is very detailed and labour-intensive. This means we are quite restricted in how many products we test.

Any anti-malware vendor is welcome to engage with us and discuss the inclusion of its products into the test suite.

Secure Windows XP after updates end

Sticking with Windows XP? Here's how to secure your system to a reasonable standard.

We'll cover:

1. How attackers work.
2. The significance this has for Windows XP users who will no longer receive updates to their operating system.
3. Free solutions to help secure your PC.

2014/01/14: This article has been updated, correcting NetMarketShare figures regarding how many people use different versions of Windows. In practical terms there is little difference.

Microsoft will soon stop issuing security updates for Windows XP.

At the same time it will cease issuing updates for its anti-malware product (Microsoft Security Essentials) for Windows XP.

This is significant because a very large minority of PC users still have Windows XP installed. Should they buy a new Windows 8 PC or can they maintain a decent level of security once they are abandoned by Microsoft?

In April 2014 Microsoft will end support for Windows XP and its free anti-malware protection. However, in December 2013 29 per cent of Windows users were still running Windows XP.

To put things into perspective 44.5 per cent were running Windows 7 and only 11 per cent were running Windows 8 and Windows 8.1 combined. These figures are provided by NetMarketShare.

Clearly such a large number of people are not going to switch to Windows 7 or 8 in the next three months.

The following article explains what the threats are for Windows XP users, how they work and ways in which users can secure their old computers without having to upgrade the operating system.

How hackers do it

There are two common ways for hackers to attempt to gain access to desktop computers.

Social engineering

The first is so-called social-engineering, in which they trick victims into running a malicious program. This program may be designed to steal information, such as passwords, from the system. Let’s call this type of software ‘spyware’.

Alternatively the software might try to further trick or blackmail the victim, perhaps by claiming (ironically) that it has detected a malware infection or by locking the PC and demanding payment for releasing the system back into the user’s control. These types of threats are called ‘rogue anti-virus’ and ‘ransomware’ respectively.

For social engineering to work the user usually has to be convinced to run a program. If they are sufficiently convinced that they need to download and run a certain program (or insert a strange USB storage device) then they will probably carry on regardless of what their anti-virus program tells them.

Some may check themselves if they see a warning like, “This file is a Trojan. We recommend you should delete it.” but clearly enough users are fooled for the criminals to continue with this tactic.

No amount of patching Windows will change this situation so, for Windows XP users, this type of threat remains as significant (but probably no worse) as before.

Software exploitation

The second method is to gain access to the system using automatic attacks. These usually involve the victim visiting a website that contains some malicious code. This code, known as an exploit, runs on the target computer and gains a temporary level of control. It uses its new-found position of power to download and install malicious software, such as the aforementioned spyware, rogue security software and ransomware.

Automatic exploits only work because there are security holes, aka ‘vulnerabilities’, in the software on the victim’s computer. Vulnerabilities can exist in the applications that come included with Windows, such as Internet Explorer; in third-party applications such as Java, Flash and Adobe Reader; and even in hardware drivers (last month researchers published an exploit for Nvidia’s display driver).

If vulnerable software is updated to make it less vulnerable then exploits are less likely to work. For example, if you are still using Java version 6.x then your system is very open to attack because there are lots of known vulnerabilities for that software. Upgrading to the latest version 7.x will help, because there are fewer known vulnerabilities in the latest version of Java.

It is neither safe nor accurate to assume that any program has no vulnerabilities at all. Usually it’s just a matter of time before someone finds a new one. If a program is popular then there is more motivation for researchers to look for security holes because they affect the most number of potential victims.

Most popular exploits

It is hard to say whether attackers prefer to exploit vulnerabilities in Windows’ own files or those belonging to third-party software but, according to an update by the security blog Contagio, the exploit kits used by criminals in recent months seem very focussed on Adobe Reader, Adobe Flash and Oracle’s Java.

There are some exploits aimed at Internet Explorer 10 and earlier, many of which could affect Windows XP users. Switching from Internet Explorer to a browser that has continued Windows XP support (such as Google Chrome, Mozilla Firefox and Opera Software’s Opera browsers), and updating all other third-party applications would be a sensible move if you want to stick with Windows XP.

Updating automatically

Microsoft makes updating Windows reasonably convenient thanks to the Windows Update service. However, this does not usually provide updates for third-party software (although it does sometimes). Fortunately there is a free application that behaves in much the same way as Windows Update but for non-Microsoft programs.

Secunia’s Personal Software Inspector (SPI) will scan your PC for vulnerable applications and can automatically download and update those for which updates exist. You can also opt to have it download the updates but wait until you instruct it to install them, and you can even have it simply scan and inform you about available updates, rather than downloading anything.

How this affects users of Windows XP beyond April 2014

If Microsoft sticks to its plans then Windows XP will no longer receive security updates after April 2014. This means that any future vulnerabilities detected in Windows XP system files and the applications that come with it will remain unfixed. This appears to be great news for the attackers, who can locate security holes and use them without fear that their activities will be hindered by an impending fix.

The solution(s)

However, this is just one facet of the situation. Third-party applications and hardware drivers will still be updateable as long as their developers continue to provide support. Additionally, certain anti-malware software, including Kaspersky Internet Security and Symantec Norton Internet Security, are capable of detecting many types of exploits and can prevent them from taking control of the system.

I put together a list of anti-malware products that will continue to protect Windows XP after Microsoft withdraws support. Most, at the time of writing, were committed to the foreseeable future.

While Java is notoriously popular with hackers, you don’t need to remove it completely in order to secure your PC. You can keep Minecraft running happily on your system but simply disallow Java in the web browser.

There are at least five free and easy ways to protect against viruses and spyware. Windows XP users won’t be able to follow point #4 (i.e. update Windows) from that linked article but the rest are relevant for those sticking with XP.

Microsoft has a tool that helps to prevent the exploitation of vulnerabilities in its own software and those created by third parties. The Enhanced Mitigation Experience Toolkit is probably a little too tricky to use for everyday users but experts and the inquisitive can download it for free.

So while it is always best to fix the problem, by patching the security hole (or uninstalling the vulnerable application if you don’t need it!), there are ways to prevent the bad guys from gaining access even though the holes continue to exist.