Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?

[JMH]

The SunSpider JavaScript performance benchmark, devised by the developers of the WebKit browser engine, is used and quoted widely as a measure of browser scripting performance. A surprising result was recently noticed by a Mozilla developer, Rob Sayre, looking at Internet Explorer 9's performance in this test. On one of the many subtests it performs, Internet Explorer 9 was finishing the subtest almost instantly.

In and of itself, that's not necessarily very interesting; several of the subtests in SunSpider are near-instant in the browser. However, it piqued the developer's curiosity. He made some minor changes to the test—changes that don't alter the result of the calculation the test performs and that, naively at least, should be treated as equivalent—and saw Internet Explorer 9 slow down considerably. He filed a bug against Internet Explorer.

Sayre's bug report was conservative—he suggested that an optimization that Internet Explorer 9's Chakra JavaScript engine was performing was fragile, and was easily disabled by minor alterations to the code that it should disregard. Coverage earlier today of the same issue was less guarded: Internet Explorer 9 was accused of cheating in the test. The allegation is that Microsoft has built a specific optimization into Chakra that detects, and bypasses, the specific code in SunSpider, but which has no other purpose. In other words, the optimization will not do anything to improve the browser's performance in any other scenario.

Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?

 

[windows7user]

I've read about this a couple of days ago. Microsoft isn't really cheating. JavaScript engine in IE9 is just a bit more intelligent, in that it can detect useless code and skip executing it. It's called dead code optimization and the article explains it. In IE9's case, the JavaScript engine has been optimized to do very well in the SunSpider JavaScript benchmarks.

The real question here is that since it is clearly shown that a browser can be tweaked to do well in certain benchmarks, how reliable are benchmarks?

 

[Windows i7 920]

I use real situations for testing, and such testing has shown Google Chrome to be the fastest for the sites I visit.

 

[Lee]

Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.

 

[Lemur]

Video cards did (and still do) optimize for benchmark tests. They're playing the same old song. It's not that big of a deal with browsers, as no expense is laid out. It's rougher with video cards, as you buy it, you own it.

 

[ken9122]

Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.

Exactly

 

[Lemur]

Who really cares? This is just plain silly. Oh Teacher, Steve is cheating again. As if no one else does it. Oh darn, but Google doesn't, and I heard that Apple is always telling the truth.

Get over it, they all have their own way of testing; this helps them look good in the eyes of those who are not technically inclined. Beside once someone has chosen a application or browser it is what they have a tendency to stay with.

Exactly

We do tend to fall in love with our browsers. And, speaking as one, I like to see it do well in the benchmarks. If it becomes painfully obvious it is lacking, I would consider switching. Different browsers, different search engines, different platforms. Ain't computing great!

 

[mickey megabyte]

when what's important is not measurable, what's measurable becomes important.

they all lie.

 

[ken9122]

Lies, damned lies, and benchmarks

This says it all.

 

[dmex]

"He made some minor changes to the test—changes that don't alter the result of the calculation the test performs and that, naively at least, should be treated as equivalent—and saw Internet Explorer 9 slow down considerably."

when you change the conditions, doesnt that change the results?

Im surprised that dev didnt run his changes against firefox, the article just says ie9's results "slow down considerably", what about other browsers?

 

[Lordbob75]

"He made some minor changes to the test—changes that don't alter the result of the calculation the test performs and that, naively at least, should be treated as equivalent—and saw Internet Explorer 9 slow down considerably."

when you change the conditions, doesnt that change the results?

Im surprised that dev didnt run his changes against firefox, the article just says ie9's results "slow down considerably", what about other browsers?

The changes were IE specific though, because only IE had this particular coding.

~Lordbob