Hi, my name is Dick Craddock and I’m the Group Program Manager for Windows Live Hotmail. “Group Program Manager” is just a fancy way of saying that I manage the team of Program Managers at Hotmail. We have the job of planning each Hotmail release and managing that release from beginning to end. We pick what features go in and write the specifications for those features. We also manage the schedule and work closely with the Development and Test teams to make sure the software comes in on time and has great quality, performance, security, and reliability.

Since this is the first of several posts on Hotmail that I plan to make over the next few weeks, I’d like to give you a bit of background on the Hotmail service – how it came to be, and what changes we’ve gone through over the years.

History

Hotmail was born on July 4th, 1996 – the creation of a Silicon Valley startup founded by Sabeer Bhatia and Jack Smith. It was one of the very first services to offer free web-based e-mail. Originally, Hotmail was spelled “HoTMaiL,” emphasizing its use of HTML for the web user interface. Hotmail became popular quickly, and by the end of 1997 already had millions of customers. Hotmail was acquired by Microsoft late in 1997 and was later integrated with another acquisition –  the web-based calendar service, Jump. Hotmail continued to grow very quickly – reaching tens of millions of users in just a few years. Today, Hotmail has provisioned well over a billion inboxes and has several hundred million active users around the world.

The Hotmail service has gone through tons of changes since 1996. When Hotmail first started, we offered free e-mail with a 2 MB storage limit. Over time, we’ve steadily increased the storage limits to 2GB and 5GB.  Today, we offer ever-growing storage, which means that you essentially never have to worry about storage limits again. (To prevent abuse, we do limit the rate at which you can increase your total storage, but if you add storage at a reasonable rate, you should never hit this limit.) In fact, we have some customers with well over 10GBs of mail in their inboxes.

Of course the user interface has changed pretty dramatically, too. Hotmail was the first to offer safety innovations like anti-virus scanning for attachments. We integrated the calendar service, built in a reading pane, and added rules, spell checking, search, web messenger, and lots more.

But over the years we’ve also made dramatic changes to the software that you can’t see. I want to talk a bit about how our software is built and the kinds of changes we’ve made over the years to take advantage of better technology, make our development team more efficient, and integrate Hotmail and Calendar with the rest of the Windows Live services.

An architecture that endures

In the last blog post, Arthur de Haan gave you a peek inside the basic architecture of Hotmail. The architecture put in place by the founders over thirteen years ago was well designed and allowed Hotmail to grow rapidly, yet we’ve made tremendous changes and improvements over time. While a lot has changed, two powerful ideas have stood the test of time and have served us particularly well:

• Build in scale units – The Hotmail service is built in “clusters” or scale units that allow our operations team to bring more capacity online in a very predictable way and to manage the service as a collection of independent sub-systems. This makes for greater reliability and scalability, and allows us to locate our computing power in various data centers around the world. Each cluster handles millions of users – we have well over a hundred clusters, and are adding new clusters routinely.
• Separate storage from computation – The Hotmail system is comprised of many sub-systems, but two of the most important systems are the Frontend and the Backend. The Frontend is set of servers that interface to the outside world. These servers do all the computation – everything from simply formatting the web pages you see when you use the service, to running the anti-virus scanners for downloading attachments. The Backend is where your data is stored. Separating these two systems has had two major benefits.
  The first benefit of separating these two systems is that, because the interface between the two systems does not change often, teams are free to innovate on both the user interface and the storage systems at the same time. So, for example, in our 2007 release, one team built a new UI with a reading pane and lots of other new features, while another team increased the storage limits by several orders of magnitude.
  The second major benefit of separating the Frontend from the Backend is that we can scale our systems somewhat independently, and keep our hardware running very efficiently. In other words, the Frontend hardware is optimized for computation speed, while the Backend hardware is optimized for storage and I/O operations.

Changing the engine while the plane is in flight

While the basics of the Hotmail system have endured over time, a great deal has changed. One of the huge challenges of running a service like Hotmail is that when we do make changes, we have to make sure that we keep the service running even while those changes are being made. It’s perhaps a bit dramatic, but I like the analogy of changing the engine of a plane that is in flight! We’ve gotten very good at making these kinds of changes and we’re able to make them quite often.

Two of the biggest changes that have been made involve (you guessed it) the Frontend and Backend sub-systems.

Starting in 2004, the Hotmail engineering team completely rewrote the Backend system, moving it off of an expensive and dedicated Unix-based storage system and onto a system that uses Windows Server and SQL Server to exploit inexpensive, commodity computers and disk drives so we can provide users with far greater storage limits than had ever before been possible. The scope of these changes was immense – we had to physically change the hardware in the data centers to new hardware running new software, and we had to move the data from the old system to the new system with no interruption of service to our customers. Of course, this didn’t happen overnight. It was a complex project that took years to complete because of the physical limitations for how quickly we could copy data onto the new disk drives.

More recently, Hotmail was upgraded to the latest version of SQL server, which, among other things, allowed us to dramatically reduce our downtime for maintenance. Hotmail is the largest SQL Server deployment in the world, with thousands of computers and tens of thousands of SQL databases.

In 2005, the Hotmail engineering team undertook a similar rewrite of the Frontend system, and rebuilt both Hotmail and Calendar from the ground up. The old software was written in C++ and Perl, and it had served us well for many years. But the team knew that we needed to rewrite the services in a modern, managed language in order to take advantage of new technology being developed by many other teams at Microsoft. So, the entire Frontend was rewritten in C# and ASP.Net, and leveraged Windows Server and the latest version of IIS – Microsoft’s web-server – to bring Hotmail into the modern world. These changes made our development team more efficient and have enabled us to share code with all the other Windows Live properties and with other teams throughout Microsoft.

Looking ahead

Over the last several years, the Hotmail team has re-engineered the service from the ground up while growing Hotmail at an astounding pace. We’ve made critical investments to get our software where we want it to be and to give us the ability to deliver great features, great performance, great reliability, and great integration across Windows Live.

In my next post I’ll talk about the work we’ve done on one of the hardest problems on the Internet: spam prevention. Until then, I hope you’ll continue to use Hotmail and to keep the feedback and comments coming!

Dick Craddock
Group Program Manager, Windows Live Hotmail

More...

Posted By: z3r010
05 Jan 2010