Browser Benchmarks: Firefox4 Vs. IE9

The launch of Firefox 4 has be heralded with much pomp and circumstance, and a great gnashing of teeth over which is better. Mozilla upped the ante over Microsoft last week, claiming to have smashed a download figure for IE9. Download figures aside, which is the best?

Mozilla's download claims are flying the flag for their new browser, which has been in development for some time (it was in public beta 6 months ago, when I last looked at browsers), a lot like their major competitor, Microsoft's IE9. However, Microsoft hasn't been as 'in your face' announcing their release, as Mozilla (in fact, I wasn't even aware IE9 had been officially released until the news articles boasting of the difference in downloads came out). Microsoft is starting to advertise now though.

So, one thing to check is how far these clients have come in the last 6 months. In addition, there are many other browsers out for Windows computers apart from the 'big 5'. In fact, the EU has forced Microsoft to not ship windows with internet explorer in the EU but instead send users to a website to download a browser (yeah, go figure) with several listed, split into two groups. The first group of 5 is visible straight away, and features the 5 'big' browsers, IE, Opera, Chrome, Safari and Firefox (in a random order) The rest are visible via a scrollbox to the side. So, to see how good they are, I thought that they should be tested too. Do the 'big 5' deserve their top spot?

As always, we're using the Q6600 test rig (HP a6535c). The specs are,

• Intel Core2Quad Q6600 Cpu (2.4 Ghz quad core)
• 6Gb DDR2-6400 Ram
• nVidia GeForce 9500GS graphics card
• Windows Vista home Premium (64bit)

As before, I'm using two main benchmarks, Futuremarks Peacekeeper, and Sunspider 0.9.1. Along with two more static tests of ability, the HTML5 test, and the acid3 layout test.

It was our biggest field yet, as 20 different browsers were tested, with two of the browsers having 3 seperate entries (as they have a variation of rendering engines) for a grand total of 24 results. Some of these browsers are not available any more, but included to allow comparison with previous tests In no particular order, they were:

1. Internet Explorer 9 beta preview 5 (9.0.7930.16406)
2. Internet Explorer 9 final (9.0.8112.16421) (DOWNLOAD)
3. Chrome 10.0.648.204 (DOWNLOAD)
4. Chrome 11.0.696.16
5. Opera 11.01 (DOWNLOAD)
6. Opera 10.70beta
7. Opera 10.63
8. Opera 10.53
9. Opera 9.64
10. Safari 5.04 (DOWNLOAD)
11. Firefox 4.0 Final (DOWNLOAD)
12. Firefox 4 Beta7
13. Firefox 3.6.13
14. K-Melon 1.54 (DOWNLOAD)
15. Avant 2010-build 131 (DOWNLOAD)
16. SR Iron 10.0.650.0 (DOWNLOAD)
17. Flock 3.5.3 (DOWNLOAD)
18. Maxathon 2.5.15.100 (IE mode only) (DOWNLOAD)
19. Flashpeek Slimbrowser 5.01.023  (DOWNLOAD)
    1. IE7 mode
    2. IE8 mode
    3. IE9 mode
20. Lunascape 6.4.4.23495 (DOWNLOAD) running the following engines:
    1. Trident (Internet Explorer Version 9.0.8112.16421)
    2. Gecko 1.9.1.17 (Firefox 3.5.x equivalent)
    3. WebKit 533.3 r56034

The results might surprise you.

Normally, I'd toss up the peacekeeper results first of all. It's a nice pretty colour graph. There's a problem though. Some of the browsers tested don't identify themselves to the test server as themselves. Slimbrowser identifies as the version of IE it's set as, while Flock identifies as Chrome7, and Lunascape identifies as the same browser irrespective of the rendering engine used. But, it *is* a pretty graph, so here it is.

Chrome's running away with it, clearly. But let's add n all the other clients that had to be tested in separate sessions, and re-label those that are mis-identified, and we get this.

Two browsers not in the public consciousness doing usually well there, although both SR Iron and Flock are based on Chromium cores (10, and 7 respectively). The gap between Opera and Chrome is growing, but Opera is still a strong second of the big browsers. Safari is coming third, just ahead of the hot competition between IE9 (and the IE9-Trident based browsers) and Firefox. Here, Microsoft has clearly stepped up its game in the last 6 months, while Mozilla has mainly gone for bugfixes. At hte bottom, we have the older rendering engines, which are just not pushing the data so well now. Lunascape actually gave an error during it's webkit-engine test (every time) which might explain it's poor result.

So, what about Sunspider? Let's see the results for Sunspider 0.91

It's quite a different story. First of all, three browsers didn't even manage to run the test, all apparantly based on an IE7 engine. IE9 dominated though, with yet another vast improvement since the first public beta of 6 months ago, which is reflected in Lunascape's Trident-based score as well, a fact Microsoft is happy to point out. Chrome and Opera are then close behind, swapping points as the versions stack up. Firefox4 follows closely behind them. Then it's the last-generation browsers, with, yet again, poor K-melon taking the last finishing spot.

There is more to the web than javascript and fancy physics based applications however. HTML5 is an ever-increasing part of the web. So, the handy test at HTML5test.com is nice to see just how much of the 'new stuff' each browser supports. Some support certain areas better than others, but an overall score, out of 400 is a good approximation. There's also a number of bonus points for going above and beyond the specs. So, here's the graph

Again Chromium based browsers lead the pack. Amusingly, Firefox4 beta7 actually scores higher than the final release, so clearly Mozilla has removed some advanced functionality over the last 6 months. Then Opera and Safari a bit back, while IE9 based browsers are still a long way back, although making up ground.

Finally, the classic test of layout, Acid3. These were in bands, and clearly identified the rendering engines. The IE-7 based browsers (avant, Mazaton, and slimbrowser's IE7 mode all scored 13. Slimbrowsers IE8 more scored 21, while K-melon scored 53, and the ancient Opera 9.64 came in at 85. Two gecko browsers next, as Lunascape's Gecko mode gets a 93, and the just-retired firefox 3.6.13 score 94. The three IE9-based browsers score 95 which the Firefox4's score 97. The rest (Opera 10.53 onwards, Safari, and the chromium based browsers, as well as Lunascape's gecko-mode all ace the test scoring 100.

So what's best?

Chrome continues to pile on the performance. It's gone through a rapid number of iterations in recent months, getting that speed boost. However, it's still memory hungry, a point Google has promised to address. There are also privacy concerns for some people, a point SR Iron promotes. Flock is also a decent alternative, based on Chromium7, and tailored more to the social networking user.

Opera also grows, although the speed battles of last summer are now behind it as it's resources can't compete with those of Google. They've also been focusing on other platforms, such as their mini and mobile browsers for smart-phones. However, they've kept their browser engine ahead of the rest still.

By far the most improved is Internet explorer 9. It's retaken the lead performance-wise from Firefox in their little battle, although it's still behind in standards. And it's not just the browser itself, as the Trident engine powered both Lunascape and Slimbrowser to similar results when that engine was used.

Mozilla has kept firefox4 pretty static over it's beta run, fixing bugs, but also appears to have dropped some segments of html5 which is puzzling. While it's a big improvement over 3.6.x it's slipping behind IE in terms of performance (even more so with Opera) and behind on both standards and speed when it comes to Chromium based browsers. Chromium browsers are open source too so the only real debate is if you need all the extra plugins that Firefox has over Chromium, as both are pretty sparse feature-wise, and how much RAM you have spare.

Safari is still plugging along, and is 'middle-of-the-pack' on just about anything. Chromium-based browsers or Opera are both better all-around choices though, and beat Safari on every metric tested.

Poor K-melon is looking a little dated now, although this version is only a year old. The engine at it's core though is Gecko 1.8.1.24 though, last seen in the Firefox 2.x browsers. An alpha version of K-Melon 1.70 was released in December, which uses 1.9.2, as seen in Firefox 3.6.

Slimbrowser is an unusual one. If you want to track behavour across several versions of Internet Explorer, it's the one for you. However, it performs well in IE9 mode. It's more of a developer's choice than one for end users. Lunascape has the same feel as well. However, the ability to actually run different engines at the same time, in different tabs is extremely useful, if only the engines were a little more up-to-date. Again, it's very much a developer's browser, but with more modern engines, this could be a real powerhouse.

The big question though is Firefox4. Are Mozilla right to boast of it's download figures? Are people downloading it because it's Firefox, and not because it's the best? The answer is almost certainly yes. Functionally, it's on a par with IE9, but may be letting the grass grow under its feet. Also, anyone running a beta version of Firefox 4 will have been automatically updated, and counted. The same can't be said for IE9.

However, this is Vista, an operating system that's over 4 years old, while it's successor, Wndows7 is getting close to 18 months old itself. I will shortly be giving the major browsers a Windows7 testing, and maybe Firefox will redeem itself there.

Data

Browser	Sunspider (ms)	HTML5 base	HTML5 bonus	Peacekeeper	HTML5 Combined	Acid3 (out of 100)
Avant 2010 b131	FAIL	32	0	1964	32	13
Chrome 10.0.648.204	335.3	288	13	7579	301	100
Chrome 11.0.696.16	305.5	293	13	7876	306	100
Firefox 3.6.13	982.4	155	4	1856	159	94
Firefox 4.0	357.0	255	9	3271	264	97
Firefox 4b7	369.9	264	9	3316	273	97
Flock	305.7	256	12	6858	268	100
IE9 Beta	417.8	104	5	2135	109	95
IE9 Final	274.4	130	5	3394	135	95
K-melon	8,998.7	32	0	550	32	53
Lunascape6 (gecko)	1,159.9	142	4	1749	146	93
Lunascape6 (trident)	286.9	130	5	3433	135	95
Lunascape6 (webkit)	538.2	208	0	1025	208	100
Maxathon 2.5.15.100 (IE)	FAIL	32	0	2020	32	13
Opera 10.53	399.5	174	4	4180	178	100
Opera 10.63	339.2	214	7	5605	221	100
Opera 10.70	324.9	214	7	5882	221	100
Opera 11.01	320.3	234	7	5909	241	100
Opera 9.64	3,631.0	67	0	1297	67	85
Safari 5.0.4	406.3	228	7	3972	235	100
Slimbrowser (IE7)	FAIL	32	0	2015	32	13
Slimbrowser (IE8)	517.6	32	0	2208	32	21
Slimbrowser (IE9)	332.4	130	5	3309	135	95
SR Iron	313.9	288	9	7658	297	100

If you liked this, let me know, and comment, or give me some suggestions.