Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?

Page 1 of 2 12 LastLast

    Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?


    Posted: 20 Nov 2010

    The SunSpider JavaScript performance benchmark, devised by the developers of the WebKit browser engine, is used and quoted widely as a measure of browser scripting performance. A surprising result was recently noticed by a Mozilla developer, Rob Sayre, looking at Internet Explorer 9's performance in this test. On one of the many subtests it performs, Internet Explorer 9 was finishing the subtest almost instantly.

    In and of itself, that's not necessarily very interesting; several of the subtests in SunSpider are near-instant in the browser. However, it piqued the developer's curiosity. He made some minor changes to the test—changes that don't alter the result of the calculation the test performs and that, naively at least, should be treated as equivalent—and saw Internet Explorer 9 slow down considerably. He filed a bug against Internet Explorer.

    Sayre's bug report was conservative—he suggested that an optimization that Internet Explorer 9's Chakra JavaScript engine was performing was fragile, and was easily disabled by minor alterations to the code that it should disregard. Coverage earlier today of the same issue was less guarded: Internet Explorer 9 was accused of cheating in the test. The allegation is that Microsoft has built a specific optimization into Chakra that detects, and bypasses, the specific code in SunSpider, but which has no other purpose. In other words, the optimization will not do anything to improve the browser's performance in any other scenario.
    Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?
    Posted By: JMH
    20 Nov 2010



  1. Posts : 91
    Windows 7 Professional x64
       #1

    I've read about this a couple of days ago. Microsoft isn't really cheating. JavaScript engine in IE9 is just a bit more intelligent, in that it can detect useless code and skip executing it. It's called dead code optimization and the article explains it. In IE9's case, the JavaScript engine has been optimized to do very well in the SunSpider JavaScript benchmarks.

    The real question here is that since it is clearly shown that a browser can be tweaked to do well in certain benchmarks, how reliable are benchmarks?
      My Computer


  2. Posts : 552
    Windows 8 Pro x64
       #2

    I use real situations for testing, and such testing has shown Google Chrome to be the fastest for the sites I visit.
      My Computer


  3. Lee
    Posts : 1,796
    Win 7 Pro x64, VM Win XP, Win7 Pro Sandbox, Kubuntu 11
       #3

    Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

    Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.
      My Computer


  4. Posts : 3,139
    Systems 1 and 2: Windows 7 Enterprise x64, Win 8 Developer
       #4

    Video cards did (and still do) optimize for benchmark tests. They're playing the same old song. It's not that big of a deal with browsers, as no expense is laid out. It's rougher with video cards, as you buy it, you own it.
      My Computer


  5. Posts : 568
    Win7 x64 Ultimate SP1
       #5

    Lee said:
    Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

    Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.
    Exactly
      My Computer


  6. Posts : 3,139
    Systems 1 and 2: Windows 7 Enterprise x64, Win 8 Developer
       #6

    ken9122 said:
    Lee said:
    Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

    Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.
    Exactly
    We do tend to fall in love with our browsers. And, speaking as one, I like to see it do well in the benchmarks. If it becomes painfully obvious it is lacking, I would consider switching. Different browsers, different search engines, different platforms. Ain't computing great! :)
      My Computer


  7. Posts : 8,398
    ultimate 64 sp1
       #7

    when what's important is not measurable, what's measurable becomes important.
    they all lie.
      My Computer


  8. Posts : 568
    Win7 x64 Ultimate SP1
       #8

    Lies, damned lies, and benchmarks

    This says it all.
      My Computer


  9. Posts : 1,289
       #9

    "He made some minor changes to the test—changes that don't alter the result of the calculation the test performs and that, naively at least, should be treated as equivalent—and saw Internet Explorer 9 slow down considerably."

    when you change the conditions, doesnt that change the results?

    Im surprised that dev didnt run his changes against firefox, the article just says ie9's results "slow down considerably", what about other browsers?
      My Computer


 
Page 1 of 2 12 LastLast

  Related Discussions
Our Sites
Site Links
About Us
Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

© Designer Media Ltd
All times are GMT -5. The time now is 16:51.
Find Us