Windows 7 Forums
Welcome to Windows 7 Forums. Our forum is dedicated to helping you find support and solutions for any problems regarding your Windows 7 PC be it Dell, HP, Acer, Asus or a custom build. We also provide an extensive Windows 7 tutorial section that covers a wide range of tips and tricks.

Windows 7: Getting a reputation: How SmartScreen looks at URLs

11 Mar 2010   #1

Windows 10 Workstation x64
Getting a reputation: How SmartScreen looks at URLs

I'd like to talk a bit about how we determine the reputation of different URLs and IPs and use this to protect against spam, phishing, and other abuse in Internet Explorer and Windows Live Hotmail.

Let's start with a bit of background. When an abuser–a spammer, phisher, or malware distributer–attacks someone, they have to do two things. First, they deliver a communication (often a spam e-mail), that entices the victim. Second, they "seal the deal" by actually selling the product, stealing the personal information, or installing the malware. (The second part is sometimes referred to as "collecting the conversion.") Dick Craddock and I have talked about some of the steps we take to block abusers' initial communications in previous posts (Fighting the war on spam, Spam, phishing, and other annoyances, and Preventing spam and phishing using e-mail authentication). I'm going to talk about some of the work we do to keep abusers from "sealing the deal."

By far the most common way abusers collect their conversions is using webpages, like the ones shown here:

Sample malware webpage

Sample spam webpage

A number of technical steps go into displaying a webpage, and the reputation systems in SmartScreen® key in on all of them. Here's a quick rundown. Consider the webpage selling medications in the figure above; to visit it you can type the URL into your web browser (although the link is probably dead by now—SmartScreen forces abusers to move quickly):

Obviously SmartScreen's reputation systems learn that particular URLs are bad—that is the first step—but we go much further. Every URL is hosted on a domain. In this case the domain is "". Abusers will often host hundreds or thousands of individually abusive URLs on a single domain. With the right evidence, SmartScreen's reputation system will flag whole domains as abusive.

URLs and domains are concepts that let humans refer to computers. But every computer that's directly on the Internet also has a numeric code, called its IP address, that lets other computers refer to it. For example, might be the IP address of the computer that's running the web server that's hosting the domain. SmartScreen's reputation system tracks these as well and will mark specific web server IP addresses as abusive. SmartScreen will also generalize to other computers "in the neighborhood" of known bad ones. For example, IP addresses are often allocated in blocks, and it's likely that the person who owns also owns and .144 and .145. We use knowledge about the way infrastructure blocks are allocated–into subnets, ASN (Autonomous System Number) blocks, the way message routing works, and more–to figure out what other computers the abusers own, and prevent those abusers from attacking Microsoft customers.

DNS servers are another key to SmartScreen's reputation system. DNS servers translate the URLs that you type into your browser into the IP addresses used by computers. SmartScreen assigns a lower reputation score to DNS servers that seem to know just a little bit too much about abusive domain names.

Making it too expensive to abuse

The point of building reputation on all of these different types of Internet infrastructure is that it costs abusers money. For example, when we find a DNS server that an abuser owns, we give it a bad reputation, and they will then need to invest in a new DNS server. When we find an IP address provider that works with abusers, we proactively find the IPs that they're registering and keep an eye on them. This figure illustrates the increasing costs that abusers incur as we dig deeper into their infrastructure.

Conceptual cost pyramid for Internet abuse

Our goal is to set up a situation where abusers don't make enough money to make it worth their time to attack Microsoft customers, where they find that getting their message in front of our users is hard, and collecting conversions is harder still.

Building and maintaining reputation

Let me now focus in on one specific piece of the reputation system behind SmartScreen: the URL-based reputation system used to fight phishing. Keep in mind that this is just one of over a dozen interrelated systems that work together to help SmartScreen do its job in protecting customers.

Conceptual architectural diagram of phishing reputation

SmartScreen's reputation systems begin with telemetry feeds: reports from end users, data from third parties, traffic from URLs showing up in e-mail, logs from our services, etc. Some of these feeds contain billions of URLs per day. Other feeds contain URLs that a third party has certified to be known phishing sites, and still others contain little more than the fact that an URL has appeared in spam e-mail messages.

Reporting phishing and malware from Internet Explorer

Reporting phishing and spam from Hotmail

But we don't assign a bad reputation based on just a single piece of feedback; any given piece of feedback may be from an abuser, from a competitor, or it may be incorrect. Instead, we use a series of algorithms that combine all the data we have to produce the most accurate and comprehensive reputation database possible. Every input feed is different, and each is handled differently, but in general, we take every URL in every feed and use machine learning to predict the probability that the URL is abusive. At a high level, this involves examining each URL for suspicious substrings (for example, the word "pharmacy" in the URL), looking up the history of the URL–its associated domain, IPs, DNS servers, routers, subnets, ASNs–and combining these into tens of thousands of potentially predictive features for the URL. We then apply models based in machine learning, which pore over these features and separate the abusive URLs from the honest ones.

Most of the time, we are confident enough in the findings of our machine learning engine that we can flag a URL as abusive based on this recommendation alone. Sometimes a URL is suspicious but we're not certain; we send many of these suspicious URLs to our analysts for final classification.

How SmartScreen reputation protects you

Conceptually, the work of SmartScreen's reputation systems results in a huge database of information about abuse on the Internet. We ship information from this database, on a near-real-time basis, into a large number of Microsoft products and services, including: Windows Live Hotmail, Internet Explorer, Bing, AdCenter, Exchange, Microsoft Security Essentials, and more. Each of these services implements some of their safety features based on SmartScreen's reputations.

In the case of Hotmail, the results are used to determine if incoming e-mail messages should be delivered to our customers. Our goal is that Hotmail customers never see messages linking to known phishing, malware, and spam sources. In other scenarios, like when a customer types the URL for a known malware site into the address bar in Internet Explorer, SmartScreen provides a visual warning.

Examples of SmartScreen reputation at work in Internet Explorer

False positives

It's worth noting that any nondeterministic filtering system can make mistakes. And, although they are rare, we take mistakes in SmartScreen very seriously, measuring them, managing them, and responding to them as quickly as possible. For more details on what to do if you think SmartScreen is making a mistake, try these resources:


SmartScreen's reputation systems bring together the telemetry, feedback, and protection of several of Microsoft's major Internet services and tools. As a result, each is safer than they would be if they had to fight abuse alone. For example, in the figure below, each color represents the size of the contribution of each different feed to SmartScreen's reputation database. Notice that no single feed accounts for more than about a quarter of the overall protection.


My System SpecsSystem Spec
11 Mar 2010   #2

Windows 10 Pro x64

yet another IE plug fro MS, thats 3 in 4 days! (4 including the bit on BBC news on saturday) is the browser ballot really hitting them THAT badly?

appropriate i think:
My System SpecsSystem Spec
11 Mar 2010   #3

Windows 2000 5.0 Build 2195

How could you call this a "plug"-in when it's built-in? It's been here since IE7, only this is better and the previous version was called Phishing Filter.
My System SpecsSystem Spec

12 Mar 2010   #4

Windows 10 Pro x64

i didnt mean plug as in fix,i meantthey really pushing the "our browser is safer than everyone elses" thing
My System SpecsSystem Spec

 Getting a reputation: How SmartScreen looks at URLs

Thread Tools

Similar help and support threads
Thread Forum
smartscreen filter
This service has been down now for me for days if not weeks, I started noticing it when started testing IE9 again and since then its not been up for me at all on IE9 and is rarely up on my IE8. The IE9 pc has default windows 7 firewall settings enabled. This machine also has windows 7 firewall...
Browsers & Mail
IE9 SmartScreen Application Reputation
More - IE9 SmartScreen Application Reputation - Softpedia
How SmartScreen looks at URLs
Getting a reputation: How SmartScreen looks at URLs I'd like to talk a bit about how we determine the reputation of different URLs and IPs and use this to protect against spam, phishing, and other abuse in Internet Explorer andWindows Live Hotmail. Let's start with a bit of background....
IE8 SmartScreen in action

Our Sites

Site Links

About Us

Find Us

Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

© Designer Media Ltd

All times are GMT -5. The time now is 08:47.
Twitter Facebook Google+