|22 Sep 2010||#1|
| || |
Firefox 4 beta loses to IE9 beta in browser speed, efficiency tests
The latest wave of upcoming changes to the world's two most used Web browsers, jointly responsible for easily three-fourths of the Internet's HTTP requests, has nearly everyone in the business rethinking the meaning of "quality" as it pertains to browser architecture. While their arguments start with the usual reminders that folks just want to see their pages load faster, before too long, they wander into dissertations about the methods architects use to achieve the appearance of loading faster. . . especially when they actually don't.
In preparing to test Microsoft's first Internet Explorer 9 public beta, released last week, and Mozilla's public Firefox 4 beta, released late last month, the advice I received most often fell into two departments: 1) Pay more attention to graphics rendering, since new browsers will be spending more time processing Web apps than just displaying pages; 2) start paying attention to how browsers utilize memory and CPU cycles. Since my smarter readers are typically right, that's what I've done in crafting my all-new browser performance test suite.
In the first round of total scores for the latest beta builds, the first IE9 beta has squeaked by the latest FF4 beta by about 8%. These scores take into account four major departments: computational ability (30%), rendering ability (30%), scalability (losing less processing ability over time as processing jobs scale up, 20%), and now, hardware utilization efficiency (20%). In past tests, as readers will recall, stable Firefox builds have typically scored around nine times the current IE8 score.
With last week's release, Microsoft has made up all of that ground, and then some. If you consider the general performance of the old Firefox version 3.0.19 (stable) as scoring a 1.000 in all categories (my new index browser), then by comparison, IE9 scores a 3.652 - better than three-and-a-half times the performance of what Firefox users were typically experiencing as recently as 2008. Firefox 4 Beta 5 posts a score of 3.377 in the same suite of tests.
What the numbers tell us
IE9 was not computationally faster across the board. It only scored a 2.077 on the JSBenchmark battery, which is even below the score for the latest stable version of Firefox (2.161 for version 3.6.10). A recent blog post by IE9 lead program manager Jason Weber, written in preparation for the IE9 beta's release, tries to spin testers' opinions away from certain brands of publicly available tests, describing and even diagraming them as incomplete. But Weber's description matched that of the original Celtic Kane benchmark, not the present test which is a complete replacement, and by the author's own admission, far more applicable to real-world workloads.
The JSBenchmark score, coupled with the blog post, gave credence to some Firefox users and other naysayers who argued that Microsoft has merely been advancing IE's scoring ability on the SunSpider test, and neglecting other departments. Is this true? To find out the answer for myself, I used a series of benchmarks that I wrote myself, for which neither Microsoft nor Mozilla have the source code. They're blistering problem-solving tests that are executed in varying workload sizes, to test for both performance and scalability - the ability to kick in more processing power when it's needed.
The news for the naysayers. . . is very bad indeed. IE9 absolutely flattened FF4 on my own DFScale battery, with a speed score of 6.590 versus 3.539. Maybe IE9 isn't the best performer in every test, but it's not because Microsoft is too focused on test numbers.
But at what cost?
As is the case with any other kind of machine ever built throughout history, when you build it to perform better, there's a cost in terms of efficiency. The Chevy Chevette may have been the single ugliest product to emerge from Detroit (way uglier than the Edsel), but it did have that fuel economy thing going for it.
So one of the most important questions we can ask when testing Web browsers, is this: In making Internet Explorer 9 a far better performer than its predecessors, did its architects sacrifice "fuel economy?" Or more specifically, does IE9 chew up more memory and processor cycles?
Say what you will about IE's performance through the years, but one advantage of working next door to the Windows architects that its own architects can use to their advantage, is building a more efficient browser. Even Firefox's most fervent supporters have historically conceded that it's a memory hog. Indeed, Firefox 3.0 (my new index browser) was a virtual black hole into which memory appeared to fall with no chance of escape.
If we score that black hole a 1.000 for CPU and memory efficiency, then by comparison, IE8 - which in several respects is more like a Chevette than a Corvette - scores a 2.845, judging from the SunSpider battery, SlickSpeed CSS selector processing test battery, and DFScale battery combined. Against most other browsers, including the stable versions of Firefox, that's actually very high indeed. This is the one scoring department where we can expect the latest, high-performance betas to fall, not rise.
Overall, IE9 beta does pay a price for its computational superiority, scoring a 2.014 in the efficiency department. But that average doesn't tell the complete story: The current stable IE8 sips memory cycles at low workloads. My latest addition to the complete test suite, adapted from The Man in Blue's animation benchmarks, coupled with processor cycle and memory consumption monitoring using Microsoft's Windows 7 Performance Monitor, shows that in low-workload animation processing, IE8 can consume less than 0.2% of available processor cycles, while Firefox 3.6.10 chomps away at 21%. The IE9 beta does consume more CPU cycles at low workloads than its predecessor, maxing out at about 4%, but that's still impressively low.
As workload increases, however, IE8 opens its throttle and starts chomping down gas like it was water. The Firefox 4 beta posts some very impressive numbers in middleweight workloads, with efficiency scores on individual Man-in-Blue heats approaching 14 (reflecting, for example, about 0.7% overall CPU cycle consumption when plotting 100 simultaneously moving objects on the HTML 5 Canvas element). But at high workloads, where IE8 can actually crash (I had to repeat some tests several times until it didn't crash) and Firefox 4 falls through the floor, the IE9 beta moves steadily along, posting numbers that are still above that 1.000 index mark.
Google Chrome's recent claim to fame has been its slick graphics and high frames-per-second numbers. My latest test confirms that claim, but it also shows the price Google pays: In the "beauty" department, to borrow Microsoft's new term for it, IE9 beta scored a 3.122 in performance and 1.832 in processing efficiency. Google Chrome 6, meanwhile, scored only 2.892 in performance and 1.461 in efficiency. So think of IE9 as a more fuel-efficient performance engine. If you do the math, this suggests that IE9 makes about 26% better use of its CPU cycles and allocated memory for overall graphics production (which, for the first time, includes SVG) than does Chrome 6.
However, Firefox 4 Beta 5 did beat the IE9 beta in the efficiency department, scoring a 2.295 overall. Memory and CPU utilization efficiency have never been Firefox's strong suits, so FF4's high scores here are very encouraging. Particularly, FF4 is learning to scale down its CPU cycle consumption with lower workloads; for example, reducing its low-workload consumption averages from 9.1% to under 1%. Firefox 4 could very well be the best behaving Firefox that Mozilla has produced to date. The problem is, it'll have more company.
The complete test suite results, including an extensive analysis of every browser in the field including Opera and Safari, is the subject of the first Ingenus Report, available soon.
|My System Specs|
|22 Sep 2010||#2|
| || |
I'm all for fast processing on my computer, but I'll be honest in that I don't really feel much difference whatsoever between IE, Firefox, Chrome and Opera and it's been like that for me over the past 2-3 years. My reasons for liking Firefox so much have to do with features coming sooner (pop-up blocker, tabbed browsing, add-on support, tear off tabs) and familiarity with the product(when you spend so much time with something, it becomes second nature.). So, for me, unless another tool comes that does something radically better, I'm not even inclined to give the other products a chance. People can rant and rave about speed all they want...but it's one of the lowest factors in the mix for me.
|My System Specs|
|22 Sep 2010||#7|
| || |
Its hardly seconds, possibly milli seconds,
I've said it before all these tests are garbage, unless everyone has exactly the same setup, ISP, and broadband details its only good for the system its tested on.
Why is that so difficult to understand, all testing needs a level playing field, variations will give varying results hence results are not a true answer.
I don't understand all the obsession, a browser is only a tool that connects you to the internet, would you buy lots of different hammers to knock a nail in.
Personally I think that any browser that needs lots of add ons is a waste of space.
|My System Specs|
|22 Sep 2010||#9|
| || |
Not sure what tests that report used, but IE9 is still slower than Minefield or FF 3.6.10 on the Peacekeeper benchmark, although I will credit the IE developers for gaining a lot of ground in the past year.thi
Something about that graph smells like fish though
|My System Specs|
|Similar help and support threads for2: Firefox 4 beta loses to IE9 beta in browser speed, efficiency tests|
|i need to delete IE9 beta and install a better browser||Browsers & Mail|
|Firefox 4.0 Beta 12 Will Probably Be the Last Beta||Browsers & Mail|
|Firefox loses browser share third month running.||Browsers & Mail|
|Opera 10 Beta Browser!!||Browsers & Mail|
|Microsoft's own speed tests show IE beating Chrome, Firefox||News|