Microsoft Rebuts Antivirus Test Failure

[A Guy]

Working in the Microsoft Malware Protection Center must be tough. Here your company is among the biggest in the world, yet a little (but well-regarded) antivirus testing lab in Germany says your antivirus software doesn't cut the mustard. Not just once but twice in the last few months, Microsoft failed to receive certification from AV-Test.


Joe Blackbird, Program Manager at the Center, explained in a blog post that failing this test does not mean Microsoft's user aren't protected.

Source

Um, ok, so since their customers hadn't encountered these particular zero day exploits, they are actually well covered...good logic

A Guy

 

[gregrocker]

Just a heads up for what it's worth: PC World once published AV rankings with a huge Norton display ad running up the side of the page.

Don't you think if there was a better AV for protection and performance we would know about it?

 

[Layback Bear]

Take a look at what anti viruses are advertised on PC Mag for sale.

 

[A Guy]

Errr, that doesn't change the message. Not saying people are not well protected by MSE, although other products managed to do well on the tests. But the reasoning on why they didn't do well is still flawed. And we do know about other AVs that perform better?

A Guy

 

[marsmimar]

I'm the first to admit I'm not the sharpest knife in the drawer. So if anyone can enlighten, please do. Also from the article:

The test gives equal weight to three elements of security: protection (keeping new malware from infesting a clean system), repair (clearing out malware that's already present), and usability (doing the job without slowing the system or falsely accusing valid programs). Microsoft did OK in the repair and usability areas but got just 1.5 of 6 possible points for repair.

And from a Techspot article that gave a bit more info about repair and avoiding false detections:

Security Essentials did fare well in removing infections from critical system areas (12 percent higher than the industry average) and received a perfect score for avoiding false detections. The software also placed third for overall usability and performance.

Microsoft Security Essentials fails AV-Test certification... again - TechSpot

Here's my confusion. How can MSE score 12% higher than industry average for removing infections if it can't detect them in the first place? And get only 1.5 out of 6 points in the AV-Testing?

 

[gregrocker]

And we do know about other AVs that perform better?

Not sure you meant that as a question but which would they be though? I've only heard this on rare occasion about Eset.

I still can't understand why every other superior tool would rise to the top here but no AV has except MSE. We do know MS's flagship OS better than anyone else.

I guess we do probably have a bias towards performance. Causing OS problems is surely a deal killer here.

 

[Layback Bear]

There you go again marsmimar thinking. You do that a lot. It's a very good question.
How does it remove a infection it can't find. Doesn't add up with my 3 brain cell either.

 

[M1GU31]

Yeah they gave comodo 2012 the same rating as mse, they recommend to use avg instead lol. I really dislike avg, and won't use that. I cleaned a system with windows defender offline which is from Microsoft last week and it removed some ransomware off a pc that had fake police alerts about having to pay a fine.I was able to clean out everything else so it does the job. My friend clean his girlfriends computer with mse also and has no issues with it and even did a scan with malwarebytes which reported it to be clean. I think there biased towards companies that have subscriptions to pay for there anti virus to give them bigger ratings so ppl will be scared away from the free alternatives and use the pay only softwares.

 

[A Guy]

And we do know about other AVs that perform better?

Not sure you meant that as a question but which would they be though? I've only heard this on rare occasion about Eset.

I still can't understand why every other superior tool would rise to the top here but no AV has except MSE. We do know MS's flagship OS better than anyone else.

I guess we do probably have a bias towards performance. Causing OS problems is surely a deal killer here.

It was not a question. In these tests, many AV programs perform better then MSE. You are looking at it from a different view. You see problems created by AV programs, so you rule them out because of that. But many, many people use them with no issues. I use Avast for instance, and it has never given me any problems, and has no effect on performance.

I will use an analogy. I am a TV tech. When people ask me which brand I would get, I am not just looking at which has a better picture, or even which are more reliable, but also how hard they are to fix when they have a problem. The majority of people have never had a problem with their TV, but I work on 8 or 9 TVs every weekday for 20+ years. How many 100's of millions of TVs haven't I seen?

I understand you like MSE because you have few, if any issues caused by it, but it doesn't mean it is the best AV program. I suspect the majority of users don't need the very best either, because their habits do not put them in a position to need that level of protection. Thus MS's explanation that these zero day samples have never shown up in their telemetry (although them having all this telemetry from their users is also a bit scary, another discussion no doubt). But if an AV program that tests better then MSE, does not give the user any problems, then it is a good choice for them. To not use them because there is a possibility they might cause problems is not the path most would choose. If they had issues, they could always switch.

I am not anti MSE, my choice of AV is not the top rated either, but since it works for me, and with no issues, I continue to use it. But the fact remains that that particular test company failed the program. That may not mean it is still not sufficient, but it does mean compared to other choices, it scored lower. A Guy

 

[jimbo45]

Hi there
C'mon guys this stuff is TOTAL and UTTER BOVINE SCATOLOGY.

Any sort of post analysis report isn't even worth using as toilet paper as a NEW ATTACK can appear at any time whatsoever rendering all the old stuff 100% irrelevant. It's the American football equivalent of Monday Morning quarterbacking.

Any A/V effectiveness comparison report is by nature of the beast totally OUT OF DATE by the time it appears and in any case only REAL TIME PROTECTION is worth considering. -- If you run a background job that tells you your computer is infected --what do you do next -- You don't know EXACTLY when it became infected or what the virus / malware has done.

Since AV software can never be 100% effective how is it people seem to believe that the removal process will itself be 100% OK -- and if the OS is infected would you trust that OS to work properly -- no IMO only a clean backup image or total OS re-install would suffice.

For Windows 8 the AV software is built right into the kernel itself -- although confusingly called Windows Defender which in W7 is rather an insipid piece of fairly useless software in W8 it's actually a re-worked version of MSE and is very effective. Also Ms updates this almost daily - whereas 3rd party suppliers have to "reverse Engineer" M/S's code to find their solution to a new threat (takes time) or use published Windows API's which any even starting hacker can exploit without any problems at all - so for Windows 8 the need for home users to have any 3rd party AV software these days is NOT NECESSARY.

For W7 MSE is still about as good as it gets for home users as MSE is being developed and strengthened all the time --it's the basis for W8.

W8 security IS important because of the next generation of W8 tablets and phones coming on stream -- smart phones currently have almost ZERO security -- Android leaks like a sieve so be VERY CAREFUL if making financial transactions by mobile --especially if using those scanning type of squares with your camera when buying services.

So while this is a W7 forum and a lot of users have no intention of going to W8 or hate it you can at least be glad of Ms's concern with W8 for mobile devices so that that MSE will continue to be an extremely important and robust product - even though the fundamental nature of security in W7 is different to W8 and unfortunately inherently not as secure.

In any case the best security is always YOURSELF - I know people who have used computers for over 40 years with no security whatsoever and have never to this day encountered a virus of any kind (wouldn't recommend that approach though).

As I've often said --for HOME users --not Corporate Lans where different issues apply the day of 3rd party A/V suppliers IS WELL AND TRUELY OVER. Especially as people move over to W8 and beyond.

Cheers
jimbo

 

[A Guy]

Sure there will always be new attacks, but don't you think that an AV programs previous track record is important as well? If an AV program has always done poorly, surely it is important info? Just as if an AV program is historically good, then at least in the present, that is indicative of what you can expect? I use Windows 7, not Windows 8, so I have my current OS to protect

A Guy

 

[jimbo45]

Sure there will always be new attacks, but don't you think that an AV programs previous track record is important as well? If an AV program has always done poorly, surely it is important info? Just as if an AV program is historically good, then at least in the present, that is indicative of what you can expect? I use Windows 7, not Windows 8, so I have my current OS to protect

A Guy

Hi there
this is one of those cases where PAST TRACK record is 100% irrelevant -- the New threat could be something that's never happened before or so works in a totally different way to previous attacks that the AV code needs to be re-written from SCRATCH. The writer of the OS in this case MS has a much better chance of finding a fix than a 3rd party supplier who by the nature of the exercise has to delve into and "reverse engineer" all sorts of Windows functions to

a) see exactly how the threat behaves.
b) find an effective cure.
c) develop and test the code.
e) ship it out of the door.

Until relatively recently Viruses and Malware worked on fairly well understood principles - even things like rootkits. I'm not saying they were easy to prevent but the base principles were mainly technical and therefore could have "Algorithmic" solutions -- so a few geeks in a backroom somewhere could devise AV software and sell it.

These days attacks include things like "drive by infections" and all sorts of things we probably don't know about. User behaviour is also a necessary component in providing satisfactory protection -- and this requires a completely different set of skills to analyse than the pure "technical solution" which can be managed by the geeks in the AV company.

A big company like Ms can analyse 1000's of "bog standard users" etc, and probably has decent behavorial scientists at its disposal so is able to devise some sort of heuristic method for predicting a users pattern and dynamically providing a way of preventing infection.

For example I'm sure you'll agree that a user who downloads 100's of files a week from torrent sites needs a different type of protection than someone who might only use the internet twice a week and logon to somewhere like the BBC's site.

That's basically my reasons for saying in the case of AV software the previous track record of companies isn't relevant in this scenario -- these types of attacks are relatively recent and whichever way you cut it there isn't enough reliable data to use as decent statistics.

If you really want to get SOME type of AV protection effectiveness in "The REAL World" you would need to get say at least 1000 random teenagers preferably -- give them laptops to use for say two weeks with no Internet access restrictions and all fitted with different AV software and then see which laptops succumbed to any viruses/malware / spyware etc.

Getting a few scientists in a lab checking what could be possible might be fine for testing say aircraft components but you need to test HUMANS in an environment where they are actually working naturally -- and they all do it differently.

The reason I mentioned W8 in this thread (it IS relevant to W7) is that W8 security will protect things like Mobile phones --these DO need massive protection --there's almost NO security on these at the moment and as the vehicle being used for this is essentially MSE then it's good news for MSE users in W7 as they will get the updates too where possible.

MSE is being developed in W8 to be as secure as is humanely possible so of course advantages will accrue in W7 too -- the point being is that unlike some AV suppliers working on a diminishing market -- W7 while excellent has probably now (apart from corporates) hit it's maximum HOME user base won't have the incentive or resources to keep working on security solutions for W7 when MSE is improving all the time and there isn't a future market for them in W8.

If they really want to do something good then these AV suppliers could make money by developing security for ANDROID smart phones --there's enough of them around.

I say also that W7 has probably hit its maximum user base for HOME users (NOT COMPANIES please note) as :

a) a lot of people are buying things like tablets and other mobile devices which need better touch type screens so are using Android or the new W8 tablets.

b) most new PC's (which although a large market is a falling one) pre-installed with W8.

Some (like they did with VISTA ==> XP) will downgrade (if "downgrade" is what it is) to W7 but the vast majority will just stick with whatever is on the computer when they bought it.

Even people on this very Forum probably at times don't use a PC nearly as much as they used to when just casually surfing the net, reading emails, using things like face book or playing music. Smart phones / tablets work fine for this -- I'm not a tablet lover but I find at times using a smart phone enough for what I need and there are days (excluding work) where I don't use a PC at all -- I can even send stuff (short --not posts like these though) to these Forums from a phone too

Please note - nothing on this thread means that I prefer W8 to W7 -- I'm just stating observed and technical facts here.

Cheers
jimbo

 

[UsernameIssues]

I'm the first to admit I'm not the sharpest knife in the drawer. So if anyone can enlighten, please do. Also from the article:

The test gives equal weight to three elements of security: protection (keeping new malware from infesting a clean system), repair (clearing out malware that's already present), and usability (doing the job without slowing the system or falsely accusing valid programs). Microsoft did OK in the repair and usability areas but got just 1.5 of 6 possible points for repair.

And from a Techspot article that gave a bit more info about repair and avoiding false detections:

Security Essentials did fare well in removing infections from critical system areas (12 percent higher than the industry average) and received a perfect score for avoiding false detections. The software also placed third for overall usability and performance.

Microsoft Security Essentials fails AV-Test certification... again - TechSpot

Here's my confusion. How can MSE score 12% higher than industry average for removing infections if it can't detect them in the first place? And get only 1.5 out of 6 points in the AV-Testing?

There you go again marsmimar thinking. You do that a lot. It's a very good question.
How does it remove a infection it can't find. Doesn't add up with my 3 brain cell either.

Maybe:

"protection (keeping new malware from infesting a clean system)"
This is done with newly discovered infections.
MSE does a poor job of this.

"repair (clearing out malware that's already present)"
This is done using relatively old infections.
All infections used for this test must be known to all AV tools being tested. The testing labs write their own scripts for this testing. A pre-infection script scans to ensure that certain settings and files are as they should be. Then the OS is infected/scanned/cleaned by the AV tool. A post-infection script scans to determine if any settings or files were missed during the repair. According to that article, MSE excels at putting things right again for known infections.

"usability (doing the job without slowing the system or falsely accusing valid programs)"
Very few false positives - partly because MSE has weak heuristics and they are slow to add stuff to their lists. Of course, what I call weak and slow, others can call conservative.


To those of you that have heard my rant about MSE before - sorry, but here it goes again:
I still install MSE on most every computer that I help out with because MSE plays nice with other programs. Weeks or months later, some of those systems get infected (sigh). And I end up cleaning up several things that MSE did not flag. If I have the time, I submit those infections to VirusTotal. Usually, many other AV tools have already flagged the infection that I'm dealing with and MSE takes weeks to start flagging it.

If I'm really curious/furious about the infection, I install MSE into a VM and infect that VM with the item that I've cleaned from another computer. This is how I came to the conclusion that MSE's heuristics are weak.

And then there were my horrible experiences with tech support for MSE (many wasted hours on the phone, remote control sessions and no resolution). Almost a year went by before I found the issue/solution on my own thru posts like this one here. It may still be an issue for XP users... I've not tested lately.

BTW, parts of MSE's code honored the environment variables and parts did not.

 

[marsmimar]

@UsernameIssues

I appreciate your input.

Since AV-Test uses a scale of 0 - 6, the average is 3, is it not?. If MSE did 12% better than the industry average in the repair category, wouldn't their repair score be higher than a 3 and not 1.5 as given?

AMTSO (the Anti-Malware Testing Standards Organization) says, in part, "Members of AMTSO have published guidelines that, for the first time, set recognized standards for testing security software. The standards have been developed and agreed to by more than 40 security experts, product testers and members of the media from around the world."

In your post you said, "All infections used for this test must be known to all AV tools being tested. The testing labs write their own scripts for this testing." Are you really suggesting that Symantec (for example) will write a script that will be detected by its competitors? Especially by a competitor that offers a less expensive AV product? Sorry, and no disrespect intended, but that doesn't make any sense to me at all. Or are you saying that AV-Test writes a script and shares it with all the other testing organizations?

I think the bottom line when it comes to AV products is real simple: using something is better than using nothing at all.

 

[jimbo45]

Hi all
Why on earth would you want to cleanse a computer AFTER it's been infected -- any AV worth the name should not allow the AV in the first place (Real time protection).

I'm amazed that people seem to have trouble getting these basic facts through their craniums (crania ??)

1) If an AFTER the event / batch report flags your computer as having got an infection - pray tell me how do you know EXACTLY WHEN it became infected and WHAT HAS IT DONE TO YOUR COMPUTER SINCE so as I said only Real time protection is any good.

2) If as obviously people believe in the statistics that some of these AV companies and their pet journalists put out it is as true as day follows night that there isn't yet ANY such thing as a 100% secure AV program.

3) That being so why on earth would you trust a Virus removal program 100% either - particularly on a computer that's been infected for an unknown period.

As I said - if you are unfortunate enough to get a computer infected you can only be 100% sure that it has been properly cleansed by a total re-install of the OS or a clean image recovery -- I certainly wouldn't trust a computer that had merely been "cleansed" by one of these AV software cleansers.

Things like MSE will detect in real time any threat (within it's capability) and show up an immediate warning notice -- at this point cleansing is fine since the infection has been trapped.

OK people still like to run reports in case stuff has got through - but if you DO get infected then I still would only trust a complete OS re-install or a recovery from a clean image - whatever security system I had on the PC.

Cheers
jimbo

 

[gregrocker]

Thanks, Bill. I think you put my blindered forums-view into much better perspective with the outside world.

I have a default reaction to those who repeatedly get infected by adding Malwarebytes real-time protection in addition to MSE which has stopped many from further infection. Do others regard this as a patch or satisfactory setup for average user? It is hard to explain why they need both because MB is not an A/V, when it's catching viruses.

I notice that the recommendations for MSE and Avast which reigned for years here tapered off to only MSE after Avast added in some crapware which required uninstall in Win7 Control Panel - more trouble than needed when you also consider they bug you for registration which MSE doesn't. After repeat calls from my friend wanting to know why they were being reminded to pay for it, I finally washed my hands of trying to dodge it's crapware and grubbing.

 

[Lady Fitzgerald]

Hi all
Why on earth would you want to cleanse a computer AFTER it's been infected -- any AV worth the name should not allow the AV in the first place (Real time protection)...

Because new viruses can get through before the AV (any AV, not just MSE) gets the definition (remember, all AVs are playing catch-up at best). An ability to use a newly received definition to detect and remove a virus that "snuck" through before the definition was received is vital.

While a fresh install of an OS is an effective way to deal with an infected OS, not all viruses lurk in the OS. They often hide in otherwise innocent files, often having attached themselves to existing files on the computer. When I first set up this machine, I had a couple of trojans sneak in while downloading M$ updates, which I immediately caught with MBAM when the updates were finished, that had hidden themselves in a couple of Word docs. I had to copy the data from the docs and paste them into new docs so I could delete the infected docs (the only practical way to get rid of the infection) without losing the data. The new docs tested clean. An OS reinstall would not have gotten rid of those trojans.

 

[DBone]

If MSE customers have to be infected BEFORE Microsoft adds a signature to prevent it, we better put a horse shoe and four-leaf clover in our pockets.

 

[Lady Fitzgerald]

If MSE customers have to be infected BEFORE Microsoft adds a signature to prevent it, we better put a horse shoe and four-leaf clover in our pockets.

Sad to say, that is how all AVs work. They have to be aware of the virus before a definition can be written for it.

You forgot the rabbit's foot, btw. Might as well have full protection.

 

[UsernameIssues]

~~~
In your post you said, "All infections used for this test must be known to all AV tools being tested. The testing labs write their own scripts for this testing." Are you really suggesting that Symantec (for example) will write a script that will be detected by its competitors? Especially by a competitor that offers a less expensive AV product? Sorry, and no disrespect intended, but that doesn't make any sense to me at all. Or are you saying that AV-Test writes a script and shares it with all the other testing organizations?
~~~

Ain't English grand?
"All infections used for this test must be known to all AV tools being tested."
Was meant to convey that the testing lab picks the infections for this test based on what the testing lab knows is detected by all products being tested. They would not share which infections are used for the test.

My comments were made without reading any of the articles quoted in this thread or researching how AV-Tests claims to do things. My comments reflect more how I would do such a test. That said, here is what AV-Tests claims about this part of the test: AV-TEST - The Independent IT-Security Institute: Repair

@UsernameIssues

I appreciate your input.

Since AV-Test uses a scale of 0 - 6, the average is 3, is it not?. If MSE did 12% better than the industry average in the repair category, wouldn't their repair score be higher than a 3 and not 1.5 as given?
~~~

"Since AV-Test uses a scale of 0 - 6, the average is 3, is it not?."
Their 0 to 6 scale is just another way of restating 0 to 100 percent. What if the industry average was to repair 0.2% of the millions of things being checked? What if MSE repaired 12.2% of the millions of things being checked? MSE would* get a 1-ish or 12.2%... [*there are 3 parts to the repair test, AV-Tests assigns a number from 0 to 6 based on the results of all 3 parts. MSE did not do better than the industry norm in all 3 areas of the repair test.]

Edit: I meant to include a comment that "repaired" is extremely subjective. We have all seen the comments about registry cleaners and how performance is not harmed by having certain registry entries to point to stuff that is not there. I suspect that MSE does not bother to clean stuff it knows does not matter.


BTW, MSE scored 1.5 for Protection and 3 on the Repair part:


The paragraph that you quoted from TechSpot had a screenshot of the Protection score right above it... maybe that is what threw you.

The 12% better at repairing claim probably came from here: