PHOBOSLAB

Blog Home

Are We Fast Yet?

Mozilla constantly asks this question, but their answer really doesn't tell us all that much. Yes, JavaScript has been getting faster and faster and the v8bench, kraken and sunspider benchmarks are a good indicator for that. Yet, they all miss some issues that are quite important for HTML5 Gaming.

One of these issues was made blatantly obvious when I finished work on the performance graph for the new debug menu in my JavaScript Game Engine: Garbage Collector pauses.

All JavaScript engines periodically run a Garbage Collector to release unused memory. In some cases this GC step can take several hundred milliseconds, in which the game comes to a complete halt. Even if that only happens every 600 frames, or 10 seconds, it's far more annoying than a game that runs at a lower, but constant framerate.

Impact Debug Menu Performance Graph for Firefox 6 – the orange spikes indicate GC pauses

GC pauses have been a huge problem with Firefox 3.6 and Firefox 6 is still the worst offender here. Other browsers have shorter GC pauses, but there's still room for improvement. To gain some more insight, I constructed a new benchmark that essentially simulates a playthrough for a game level.

Give it a try at HTML5-Benchmark.com.

Now, to be clear, this benchmark is very “opinionated”. It's by no means a complete benchmark for all HTML5 features, but only tests one very specific use case: smooth running games rendered with the <canvas> element. It's a very real use case though; not an average for hundreds or thousands of small tests.

The score for the HTML5-Benchmark takes the total time the browser spent rendering frames and a big penalty for pauses into account (the exact formula is 1000000/(sqrt(totalTime) + lagTime * 0.1)). The benchmark automatically runs at a “reasonable” screen size - i.e. a screen size that makes sense for a game on the particular device. Thus, the final score is a real indicator for the browser's ability to smoothly run HTML5 games.

HTML5-Benchmark Scores

A score of about 1500 means the game is playable. Note that the benchmark actually runs a bit slower on mobile devices than the game itself. This is because the game utilizes a special pre-rendered background mode on mobile devices, but doesn't do so in the benchmark to keep the scores comparable.

It's almost comical how bad the Android's Browser performed in my tests: the Samsung Galaxy Tab 10.1 is actually slower than a four year old 1st gen iPod Touch. Firefox 4 on the same tablet performs a lot better though.

The desktop version of Firefox 6 still suffers a lot from the occasional GC pauses. This makes Firefox the last choice if you want to enjoy HTML5 games. The performance decrease from the stable Chrome 13 to the dev version of Chrome 15 is quite strange too. It may have to do something with the new hardware acceleration for Canvas' 2d context.

Impac Benchmark

There are some things that the benchmark can't measure. Opera for instance has a very steady timing and never misses a frame. On my machine however, the benchmark still looks choppy - probably because the benchmark's timer isn't synchronized with the refresh rate of the display. Opera essentially renders frames that it never displays. The new requestAnimationFrame() could fix this, but it hasn't arrived in all browsers yet.

IE9's smoothness on the other hand is remarkable. Of all browsers and systems I tested, IE9 subjectively produced the best results. It seems to precisely synchronize JavaScript execution with the refresh rate: the benchmark runs at exactly 60 FPS on my machine, even though the timer is scheduled every 16 milliseconds, or 62.5 FPS.

This leads me to ask a few questions: shouldn't browser vendors concentrate on fixing setInterval() first, before implementing requestAnimationFrame()? And shouldn't it behave more like setInterval() instead of setTimeout()? That is – I don't want to call a function to schedule each and every frame I want to draw, but just call it once and let the browser schedule the frames.

requestAnimationFrame() looks good on paper, but it doesn't actually produce visually better results than setInterval() in those browsers that implement it.

On my machine, Chrome 13 and Firefox 6 awkwardly jump between 120 and 60 FPS when using it (it's disabled in the benchmark) and even without GC pauses there are noticeable hiccups, while IE9 is able to produce perfectly smooth animations without it. At the moment requestAnimationFrame() is truly worthless for games.

So, no – we aren't fast yet. But we're getting there.

Thursday, August 11th 2011

49 Comments:

#1Gornova – Thursday, August 11th 2011, 18:24

On windows xp sp3 with Firefox 5 my score is 677, a little bit laggy sometimes.

One note: have you tested with Firefox 6 ? Do you mean Firefox 5 ?

#2tylermwashburn – Thursday, August 11th 2011, 18:28

Did you run these tests with other programs running along the browser, or did the browser run alone?

#3Dominic – Thursday, August 11th 2011, 18:53

I did test it with the current Firefox 6 beta.

I rebooted my system before each run. The browser was the only program running at the time.

#4ponce – Thursday, August 11th 2011, 20:29

Another much needed rant! I've come to optimize everything in my Javascript programs for results largely slower than a 486 from 1995. I'm tired of pooling, hoisting variable out of loops, hoisting array lengths and breaking encapsulation, without getting the nice stuff in Javascript (if something like even exists).

#5 – Adam – Thursday, August 11th 2011, 21:03

Question - will this be any different using WebGL? (in terms of Firefox, which does support it)

What I mean is - what if you would switch to WebGL in the rendering level in Chrome/FFX to draw 2D sprites, will it be faster, just as bad, or worse?

#6Geoffrey Sneddon – Thursday, August 11th 2011, 23:22

What version of Opera were you testing using? Every version since 10 has claimed to be 9.8 in the UA string (with the addition of Version/11.50 or whatever as the real version). I presume you're testing 11.50 — the latest stable release. Would be interesting to see how recent snapshots, available from <my.opera.com/desktopteam/blog/> do.

#7 – Eric – Friday, August 12th 2011, 08:08

Have you tried IE10 PP2? I am getting the best results with it (6958). Even better than IE9!

#8Ric – Friday, August 12th 2011, 10:41

You should call this Canvas Benchmark or similar. The media is already in a muddle with what the hell HTML5 is and this'll only confuse matter more. According to this site on my machine IE9 is the greatest HTML5 browser on the planet.

#9Dominic – Friday, August 12th 2011, 14:29

@Geoffrey Sneddon: Yes, you are correct; I fell for that trap. I actually tested Opera 11.50, not 9.8.

@Ric: I thought a while about the name, but in the end decided for the catchy, "incorrect" one. And IE9 is indeed one of the best browsers to enjoy Canvas games on - it's one aspect of HTML5 that I care about.

#10 – Matthias – Friday, August 12th 2011, 15:54

Thank you so much! This will save me plenty hours of debugging time for the reason why my own little canvas game feels so "stuttering" and your benchmark shows those "render lag spikes" in a absolutely obvious way.

#11Matthew Brunelle – Friday, August 12th 2011, 16:39

This is amazing. Its funny I've always been running the dev version of chrome (6398 on MBP 8,2), but I have to remember that not everyone is a crazy browserphile like I am. Actually there aren't and developers that are...

#12 – louisremi – Saturday, August 13th 2011, 21:26

I am not experimenting the 120fps bug with requestAnimationFrame
This really shouldn't happen. Could you try to produce a reduced test case?
Thanks

#13Seth Ladd – Monday, August 15th 2011, 20:22

Just got Score: 4766 Total CPU Time: 33.77s, Total Lag: 261ms on Chrome 15.0.849.1 dev on my Mac 10.6.8 w/ an i5

#14 – Phillip Grønvold – Tuesday, August 16th 2011, 04:01

Try Opera Mobile 11.1 on the Galaxy 10.1. It scores 2704 on my Galaxy SII Android phone. (Disclaimer: I work for Opera)

#15Andrea Doimo – Sunday, August 21st 2011, 10:36

7313 with Chrome 14.0.797.0 m and XP !
:D

#16 – Nicki – Monday, August 22nd 2011, 10:39

2389 Total CPU Time 147.40s, Total Lag 346ms with Firefox Mobile 6 for Android on Samsung Galaxy S 2

#17 – Garth – Saturday, August 27th 2011, 07:59

Score: 7547

Total CPU Time: 17.56s, Total Lag: 0ms
Internet Explorer 9
Nice benchmark by the way.

#18 – Mark – Sunday, September 4th 2011, 00:20

Mozilla is working on reducing GC pauses.
The first thing they seem to want to do is turn their GC into an incremental GC. This means they will set a limit on how long they can GC (say 1ms), but GC more often to compensate.
bugzilla.mozilla.org/show_bug.cgi?id=641025

They are also working on doing the slowest part of the Garbage collection in parallel for a consistent 20-40% reduction in GC pauses. bugzilla.mozilla.org/show_bug.cgi?id=638660

Later on (because it requires more work than the above 2 things) they will switch to a Generational GC that collects less often and may even collect concurrently without stopping the javascript execution (Like what the Java VM does).
bugzilla.mozilla.org/show_bug.cgi?id=619558
wiki.mozilla.org/JavaScript:GenerationalGC

More work:
wiki.mozilla.org/JavaScript:GC

BTW, on a Firefox 9 (Linux, 32bit) nightly the game ran at a perfect ~63FPS without any noticeable GC pauses, but the score was lower than Chrome which ran at 45FPS with GC pauses..... :|

#19Björn Nilsson – Tuesday, September 13th 2011, 09:06

Great points, especially about setInterval and syncing it to the screen refresh rate. Props to Microsoft for fixing this.

The only win I ever saw with requestAnimFrame was that animation is halted in non-active tabs. The promised smoothness inprovement was absent for me to in all browsers I tested.

#20Nafeez Ahamed – Saturday, September 24th 2011, 06:57

setImmediate is the answer , IE10 :)

#21 – Łukasz – Saturday, September 24th 2011, 10:01

Score: 2505 Total CPU Time: 126.34s, Total Lag: 438ms
Opera Mobile 11.10 on Xperia PLAY

#22 – Dan – Monday, September 26th 2011, 23:28

FYI: Firefox 7 (which launches tomorrow, September 26th) invalidates much of this article. Chrome CGs *VERY* frequently, Firefox 7 does CG very intelligently, and much more often, to avoid the negative performance outcomes stated in this article.

Sorry to crash the party, but the tales of CG performance issues in Firefox are apparently only going to be valid for another few hours ;)

#23 – enolive – Monday, October 3rd 2011, 03:56

Score: 4531
Total CPU Time: 43.36s, Total Lag: 125ms

on Ubuntu 11.04 x64 with Firefox 7. I didn't notice any of the GC lags I had on previous Firefox versions.

#24Alex – Monday, October 3rd 2011, 05:54

strange... Debian here. My HTML5 game (in dev) runs around 60 fps in Chrome, and this is very stable (no noticeable GC locks). But FF7 is worse than ever, not slow, but too unstable (from 50 to 90 fps). I normally don't care much as I fix game dynamics depending on current FPS. What really suck is that FF misses some mouse move events due to this (VM too busy at 90Hz). Before FF7, it was a bit better, but now, it's just not acceptable. For the time being, I decided to switch back to an old setTimeout() method which is a bit better than mozRequestAnimationFrame() !! What the hell are they doing?! I used to love FF, but now, it is really getting on my nerves.

I hope they will fix that soon, I'm very close to recommand Chrome over FF for using my stuffs...

#25 – Ian – Wednesday, October 12th 2011, 06:41

I had some strange results on my machine. IE9 scored about the same as it did in your test case (roughly 6400) and performed exceedingly smoothly. Chrome looked visually choppy yet scored around 8100. It almost appeared as though that it counted every tiny bit of stutter on IE9 as lag (which totaled ~330ms) but neglected to do the same for Chrome (which gave me a result of ~77ms, which seemed very low for how it looked).

Firefox 7 still doesn't perform particularly well. It generally holds 50-60fps, but it lags significantly more. I think it finished with about 1200ms lags total, but I stopped paying attention after it crossed the 1 second threshold. Its final score was around Firefox 6's -- roughly 4500 on my machine.

For any curious parties, my 4th generation iPod Touch scores 1895, putting it in like with the iPad. Sadly, I was unable to get the test to start on my Mango-equipped Windows Phone (the Impact logo appears and appears to go through, but after that goes away the canvas stays black; this happens when I use either the desktop or the mobile page), so I can't provide any mobile IE9 results.

#26PotHix – Wednesday, November 2nd 2011, 22:13

Æ!!

Really interesting. I just ran this benchmark on my Nexus S with the default android browser and got:

Score: 1076
Total CPU Time: 433.91s, Total Lag: 2704ms

A lot of lag, but a good score compared with your benchmark.

Trying on the Firefox Nightly for Android I got this results:

Score: 1237
Total CPU Time: 196.07s, Total Lag: 3659ms

Just for the record. ;)

#27guitar tuner – Wednesday, November 9th 2011, 23:38

indeed, let's hope Zynga's recent announcement to focus on HTML 5 will help things along. www.informationweek.com/news/development/web/231902152

Curious to see what that open source audio tool will yield.

#28Peter – Monday, November 28th 2011, 23:20

Are you going to release this benchmark on github for example for open source? if not, i'm suggesting you a partnership in which we will create an app for social networks like facebook and vkontakte in which users can run this test inside app window ...

we will get a big amount of data for analyze on hardware + earn some money on advertising,

#29 – a raccoon – Wednesday, November 30th 2011, 07:14

Request: Please add a "%" score of test-time over lag-time, with 4 or 5 decimal places, to your html5-benchmark. Eg, if the test took 500.3 s with 51234ms lag (51.234s), a value of "9.765% lag" would be displayed.

#30 – Windows Phone User – Tuesday, January 31st 2012, 14:59

Not working on Windows Phone...

#31 – WanderingNewbie – Sunday, February 5th 2012, 02:45

I just ran this on my late 2007 Toshiba Satellite A215-S4757 Notebook

On IE9, FF10, GChrome16 and Opera 11.61

The results did surprise..and hurt my feelings! =P

I ran them 3-4 times each. Reloading the page each time.

IE9 (Smooth, FPS usually around 40~50)
Score: 2997~3172 (2026-It was really slow the first time!!)
TCPUTime: 78.67s~76.05s (98.12s First time)
TLag: 535ms~369ms (1803ms First time)

FF10 (Laggy, FPS usually around 30~40)
Score: 1523~2362
TCPUTime: 82.01s~73.13s
TLag: 3701ms~1530ms
(Note: FF10 kept getting faster as I kept checking it, unlike other browsers.)

GChrome16 (Fast!!! But lags a few. It tries to be at 60 FPS full time.)
Score: 2828~3191
TCPUTime: 57.79s~52.83s
TLag: 1132ms~798ms

Opera11.61 (Just installed this for this benchmark.)
Score: 3184~4111 (This is not a typo...)
TCPUTime: 66.08s~54.44s
TLag: 589ms~99ms

I'm very surprised of this...My Browser of choice is FF10.
(The slowest!!!)

After I read this post I wanted to check the speed of IE9 and FF10 and compare. Then I decided to check Chrome and Opera.

Opera beat them all right now. It ran great.
It was unexpected for me, to say the least.

How is IE10 running on your machine compared to IE9?
I don't have it, but I would like to know.

#32 – Vincenzo – Tuesday, March 13th 2012, 19:12

code.google.com/p/chromium/issues/detail?id=117880

#33RIUM+ – Thursday, May 10th 2012, 05:29

It seems that in the 9 months since you released this benchmark, some of the major browsers have done a lot to improve their scores. Is there any chance you could test them again & upload a newer set of scores so we can see where things currently stand? :)

#34 – FTwin8 – Friday, July 27th 2012, 14:08

Hmm... Score 6927 with windows 8 and IE 10

Hardware

amd phenom x4 965 @ 3Ghz

NVidia gtx 570

4gig of 1333mhz ram

#35 – FTwin8 – Friday, July 27th 2012, 14:10

Oh and 0 lag & fps 59 - 60 and response time 2-4ms

#36 – mamooth – Tuesday, August 28th 2012, 18:18

Benchmark scores from my desktop PC:

Chrome 21: 3881
IE9 (64-bit): 2130
Firefox 14: 3135
Firefox 15: 3833

Firefox 15 added an Incremental Garbage Collection feature, thus the big improvement.

All I had was 64-bit IE9, and it's slow. People using 32-bit IE9 should get much better results.

#37 – Zaoldyec – Monday, September 3rd 2012, 05:37

ie10: 7778 :D

#38 – Zaoldyec – Monday, September 3rd 2012, 05:41

chrome 21: 5323 :X

#39 – ADIGIOIA – Monday, September 17th 2012, 02:14

11274 with IE 10

#40 – ADIGIOIA – Monday, September 17th 2012, 02:15

and my computer is 5 years old (final build of IE 10 and windows 8 pro final build)

#41 – William – Thursday, January 24th 2013, 03:12

computer running an i5 quad core lynfield with a gtx460 on ie10.

anywhere between 9000-10500 compared to 6500 on firefox and 7000 on chrome.

#42Kevin Roast – Wednesday, March 20th 2013, 12:29

There is a new HTML5 Canvas Benchmarking tool you may be interested in here: www.kevs3d.co.uk/dev/canvasmark/

#43 – Edgar – Thursday, March 21st 2013, 17:59

windows 8 and IE 10: 11453 =]

#44 – Luke – Saturday, April 5th 2014, 02:50

I tested on the xbox one and got a stable 67 fps I thought that was kinda weird. Also my score was around 3500.

#45Tahamina – Tuesday, August 5th 2014, 06:20

Hay, really great blog. I think it is more informatics and useful site. All visitors are benefited to read this sit’s article. All article is Unique and relevant. I like this kind of blog. So I am searching by <a href=”bouncycastlehireliverpool.jimdo.com/2014/05/18/why-you-thought-to-hire-a-bouncy-castle-on-your-party/”> kids party</a> and I find this site. Thanks for admin who share that’s.

#46inflatable hire – Sunday, August 10th 2014, 06:12

Hay, really great blog. I think it is more informatics and useful site. All visitors are benefited to read this sit’s article. All article is Unique and relevant. I like this kind of blog. So I am searching by <a href=”bouncycastlehireliverpool.pen.io/”> inflatable hire</a> and I find this site. Thanks for admin who share that’s.

#47inflatable hire – Sunday, August 10th 2014, 17:01

The right phases with more interest will compile your right force phases for good root matter and it produces active natures. <a title="inflatable hire" href="cityofbonesbook.com/">inflatable hire</a>

#48murri – Monday, August 18th 2014, 21:42

7478

dat.cat/oyeah.png

#49Piko – Tuesday, October 21st 2014, 15:02

Linux mint 17 on Firefox 28 in LiveCD:

Score: 9172
Total CPU Time: 11.02s, Total Lag: 40ms

PC is :
Procesor: I3 3240 3.4GHZ
4 GB ram
G card: Xeon E3-1200 v2/3rd Gen Core Processor.

I'm suprised...

Post a Comment:

Comment: (Required)

(use <code> tags for preformatted text; URLs are recognized automatically)

Name: (Required)

URL:

Please type phoboslab into the following input field or enable Javascript. This is an anti-spam measure. Sorry for the inconvenience.