Web Browser War: A Benchmarking Test

Posted: March 23, 2011 in English Version, IT & Computer Science
Tags: , ,

This post gives you a benchmark result on main web browser as of today. I’ve done some benchmarking for 5 web browsers on my laptop, or my Netbook actually. I’ve done benchmarking on 5 main web browser, each of which claiming to be modern browser and the fastest one among the others.

So why am I all in sudden make this benchmarking?? The main background is the release of Internet Explorer 9 and Mozilla Firefox 4. The other reason is that I have no job to do right now. And the main reason is that I like comparing things before I decide which product I’d buy or I’d use.

So this is it, for my big curiosity, and for the fact that I don’t have many jobs to do these couple of days, here you are the benchmark.

I’ve done this benchmarking on my MSI Wind U230 Netbook. Here is the spec:

Processor           : AMD Athlon Neo X2 (Dual Core) L335 @ 1.60 GHz

RAM                     : 2 GB DDR2

GPU                       : ATI Radeon HD3200 Integrated GPU

GPU Driver        : ATI Catalyst 9.10

Display resolution    : 1366 x 768 @ 60 Hz Refresh Rate

Operating System    : Windows 7 Ultimate 32 Bit

I tested 5 main web browsers, the latest stable / release version as of the day I do the benchmark.

  • Apple Safari 5.0.4 for PC (Windows) Build 7533.20.27
  • Google Chrome Stable Version 10.0.648.151
  • Internet Explorer 9 32-bit RTM Version 9.0.8112.16421
  • Mozilla Firefox 4.0
  • Opera 11.01 Build 1190

Benchmark suites I used are as follows:

Along with that theoretical benchmarking, I also compare all those browsers in real world performance. I opened 23 tabs of webpages and then I take a note on how much RAM, how much Page File, and how many CPU Percentage they use.

For unbiased benchmark, I make some standard condition prior to / while executing benchmark on each browser:

  • Cleaning Personal Browsing Data on every browser prior to benchmark execution. This include History, password, Cookies, Cache, etc.
  • All Add-Ons / Extensions are disabled.
  • There is no any software running – neither on foreground nor on background – while I run the benchmark. Well, actually there is software running on background, such as Anti-Virus, Firewall, Download Manager (but it’s not downloading anything), and Modem Software to maintain my Internet connection… I even don’t open Music Player, Office Application; even I don’t open Notepad or Task Manager.
  • For each test, only one browser, one window, and only one tab was opened.
  • I ensure my broadband connection is in EVDO-Rev-A Connection with speed at least 300 kbps. Anytime it droped to National Access Connection (which has limited speed connection about 128 kbps), I disconnect my internet and reconnected it until I regained EVDO-Rev-A Broadband Access.

So, this is my bench mark results.

  1. ACID3 Test

    Acid3 test is used to test a browser regarding its compliant to Web Standard Project. It checks the performance of a web browser regarding the issue of following certain selected elements from web standards, especially relating to the Document Object Model (DOM) and JavaScript.

    The results of Acid3 test I run on my laptop are as follow.

    Internet Explorer 9 and Firefox 4 failed in the test. It means that both of them are not compliant to the Web Standard Project, at least up to some level of standard. And interestingly enough, both of them are also claiming to have same feature: Hardware Acceleration (Enabled by Default).

    I don’t know what exactly this means in real world performance. But I could safely say that since both of them are not compliant with the standard, they – in some degree – will wrongly display web page. It could be fonts, layout, SVG Graphic / SVG Font, etc.

  2. Sunspider Javascript Benchmark

    Sunspider Javascript Benchmark is quite famous for benchmarking javascript performance of a web browser. But if you see that the test is held in a sub-domain of webkit.org, you could suspiciously think that this test is optimized for browser with Webkit Engine as its core layout engine. Browser such as Apple Safari and Google Chrome (as well as Chromium Browser and all of Chromium-Based browser) are using Webkit Engine. So, you might be thinking that Google Chrome and Apple Safari would win in this test… But, the result is surprisingly contrary to our (or my) thought. Google Chrome in fact lose in this test. And Internet Explorer 9 is definitely winning the Sunspider’s competition.

    So is it Bad?? Is result showing more than 700 ms bad?? Well, it might be bad, but might be not either. I’ve test Mozilla Firefox 3.6.15 before, andthe result is still more than twice as slower : 1894.8 ms. So, while a value more than 700 ms is slow enough, but it is nearly unnoticable in actual real world web browsing.

  3. V8 Benchmark Suite Version 6

    This benchmark is developed by Google and optimized for Google Chrome (and of course any Chromium-based browser too, I think). And the test I’ve run confirm this. Google Chrome has quite high point in this test. However, the throne for this test goes to Firefox 4.

  4. HTML 5 Test

    Well, despite the name, this test does not measure performance of the web browser regarding html 5 web page. It simply indicate the completeness of feature that support the upcoming HTML5 standard and related specifications. Hence, the higher the score does not imply the better performance. It only shows that the higher score mean the more complete HTML 5 feature & support. However, as HTML 5 is always talked everywhere and anytime, I of course prefer a web browser with more complete feature, I.e. higher point in this test.

  5. FishIE Tank Benchmark

    I take this test to measure how well hardware-accelerated feature implemented in web browser. For those who doesn’t know, hardware acceleration is a technology used by web browser to utilize not only CPU, but also GPU (or VGA Card). Until the moment I write this post, there are only 2 browser (stable or release version) that have this feature: Internet Explorer 9 and Firefox 4.

    Even though other browsers don’t have this feature, I tested them anyway. I just want to see how browsers with this feature are compared to those without this feature.

    FishIE Tank is a benchmark developed by Microsoft. The test measures the ability of the web browser in rendering fish-generated-moving-object and counts the fps could be showed by web browser. And the result of course is not surprising: Internet Explorer 9 wins the battle.

    Hardware-Acceleration feature could have been very helpful in the test. I mean, a browser with hardware-acceleration-feature enabled, should have a high frame rate. In this case, Internet Explorer 9 and Firefox 4 should have reached a high frame rate. While IE9 capable of running the test up to 1000 fish, Firefox 4 can only run the test up to 100 fish. Firefox 4 seems to have a little frame rate here. It even nearly is the same as Google Chrome which does not have Hardware Acceleration feature.

  6. Futuremark Peacekeeper Browser Benchmark

    Peacekeeper is a benchmark suite developed by Futuremark Corporation. It does not have any affiliation to Webkit Project, Google, Microsoft, Mozilla Foundation, Opera Software ASA, or any other web company. It is not optimized for any web browser. This make the benchmark is regarded as unbiased benchmark (at least some people said so).

    And the test results show that Google Chrome gets the highest score.

  7. Realworld Benchmark

    The main objective for this test is measuring CPU, Page File, and RAM usage of each browser under certain load.

    For the test, I opened 23 pages on every single browser. No other software is running, not in background and not in foreground either. (except of course Security software such as antivirus, firewall, etc. However, all of them are in idle state).

    Webpages I select for this test is based on my interest and some consideration as follows :

  • Containing some level of complexity
  • Implementing Html 5
  • Audio content
  • Video content
  • Flash content
  • Frequency of access (I am very frequent accessing Detik, Kompas, Republika, Kaskus, and Wikipedia)

So here you are the list of 23 websites I open for the test:

  1. HTML page I created containing links for the rest 22 webpages.
  2. Google & Login
  3. DetikCom
  4. Kompas Online
  5. Republika Online
  6. Engadget
  7. Ars Technica
  8. You Tube Search Blue Stone (Electronic-Pop Musical Band Project)
  9. You Tube Playing Déjà Vu from Blue Stone
  10. Facebook Home Page of me
  11. My Blog Page at WordPress
  12. Google Image Search for keyword JAKARTA
  13. Bing Image Search for keyword JAKARTA
  14. Yahoo Image Search for keyword JAKARTA
  15. Kaskus Thread for thread “Gambar Unik dalam Microprocessor” and expand all the picture embedded there
  16. Neurodisc Record Homepage
  17. The Killers (Band) HTML5 Home Page
  18. Wikipedia (English) page for Indonesia
  19. Supernatural (TV Series) Official Page on CW TV
  20. SyFy TV Channel
  21. Google Maps with Google Earth Layer centered on IKPT Building
  22. My Picasa Web Album
  23. Google Reader What’s Hot in Google

After fully opened all the pages, I replayed all the video & audio content in the browser for maximum load of RAM, Page File & CPU Usage.

After I was sure that maximum load already reached, I write down the CPU, RAM, and Page File Usage.

The final results are as follows.

 

It’s obvious that Internet Explorer 9 have the worst hardware utilization. It uses 100% of CPU, highest Page File and RAM Usage. Surprisingly, Apple Safari is the most hardware-friendly. However, unfortunately, Apple’s Support (and developer’s as well) for Safari in PC (Windows) is not as well as in MacOS (at least, I couldn’t find XMarks extension for Safari on Windows).

 

Overall Comparison

Comparing all the benchmark, here I give you a handy comparison table. Blue-Bold Font means the best in the certain benchmark, while the Red-Bold is the worst.

Synthetic benchmark gives a non-conclusive result. However, Google Chrome wins in 2 benchmarks and pass ACID3 test. Internet Explorer 9 also wins in 2 benchmarks, but it failed in ACID3 test. Opera and Firefox win in 1 benchmarks, but while opera passes ACID3 test, Firefox failed. Apple safari does not win at all in any benchmark. It is only capable of passing ACID3 test.

So statistically, Google Chrome is the best browser in term of synthetic benchmark.

How about real world benchmark?? The following table gives you hardware utilization for each browser. Please note that the lower the value, the better the browser utilize hardware.

In this real world test, while it shows a non-conclusive result on which the best browser is, it does give a conclusive result on which the worst browser is. Internet Explorer is obviously the worst in hardware utilization. It is devouring hardware resources, taking more RAM, Page File, and CPU usage among other browsers.

In this real world test, the best browser seems to be Apple safari. It’s a bit surprising for me. Google Chrome – which is also using Webkit Layout engine – does not perform as well as apple safari 5. Especially in CPU Utilization and RAM Usage (you can ignore Page File usage since it is only differs by 120 MB which is considered as not very much).

Verdict

So, which one is the best??

It seems like no one. As each browser seems win in certain benchmark but lose, or even the worst in other benchmark.

In term of ACID3 Test, just don’t choose Internet Explorer 9 or Firefox 4. Failing in this test means that it is not fully compliant with Web Standard Project. Even though Mozilla criticize ACID3 test, it is still give you a chance that certain website would not displayed properly on Firefox 4 and IE9.

In term of Javascript Performance, Sunspider, V8, and Peacekeeper seems give you a non-conclusive result. Sunspider result differs only by 256.2 ms second between the fastest and the slowest score. I think it is a small margin to even noticeable in real world performance.

I see that Futuremark PeaceKeeper is the most comprehensive test among all (It takes nearly 5 minutes to run a single set of browser benchmark, while Sunspider and V8 only take no more than 1 minute). It is also the most un-affiliated to any browser company.

So, while V8 and Sunspider is not conclusive, I think you can safely say that PeaceKeeper is quite good unbiased benchmark. And Google Chrome win in this benchmark by difference at least 702 score. I can say it is 22% faster / perform better than Opera, and even 85% faster / perform better than Firefox 4.

In term of Html 5 Compliant & completeness, it’s obvious that Google Chrome has the most of HTML 5 support.

In term of Hardware Acceleration, Internet Explorer 9 is the beast. But remember, FishIE is developed by Microsoft, hence it could be tweaked and optimized specially for IE9. It is quite obvious since Firefox 4 which is claiming to have Hardware Acceleration feature performs not much better than other browser. Firefox 4 even has close frame rate to Google Chrome 10 which does not support Hardware Acceleration.

And last, in term of hardware & source management, just don’t choose Internet Explorer 9.

So, which one is the best?? Well, it’s kind of trade of… But personally, I’d say Google Chrome is still the best.

It might be my subjective preference. Yeah, all the words I write here could be subjective view of me. But the numbers and charts I display above are not subjective. They are just what they were on my Netbook. So, even if I say Google Chrome is the best, you can have another opinion by analyzing those numbers and charts. Just don’t say that I set up the test or make up the data. I was not making those data. They are what they were as I tested them on my Netbook. And it takes me 3 days just to finish this benchmark and  post!

Damn, what am I actually doing?? J

I hope everyone who read this would be pleased.

End Note

I don’t have any educational background in IT. I am a professional Process Engineer specialized in Oil & Gas Engineering & Construction. But I do have interest in IT and read a lot many IT stuffs. This benchmark might be not well written, but I tried to run the test as objectively as possible. While the opinion on the text above might be personal, the numbers and charts are not.

  1. Screenshots

Acid3 Test Screen Shot

V8 Benchmark Screen Shot

HTML5Test ScreenShot

Peacekeeper ScreenShot:

Comments
  1. och says:

    Anak IT ya mas??? pantesan.. Aq g ngerti sm skali pembahasane…. Dadi intine..seng apik tetep chrome. Gitu??

    • thing2think says:

      Bukan anak IT sayah. Khan udah dijelasin di tulisan di atas kalo saya bukan anak IT :-D… Yup. Google Chrome is still the best.

  2. Luke says:

    Thank you for conducting this test. I really appreciated the depth and layout of the results.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s