Are We Fast Yet?
Performance Graph for Firefox 6 – the orange spikes indicate GC pauses
GC pauses have been a huge problem with Firefox 3.6 and Firefox 6 is still the worst offender here. Other browsers have shorter GC pauses, but there's still room for improvement. To gain some more insight, I constructed a new benchmark that essentially simulates a playthrough for a game level.
Give it a try at HTML5-Benchmark.com.
Now, to be clear, this benchmark is very “opinionated”. It's by no means a complete benchmark for all HTML5 features, but only tests one very specific use case: smooth running games rendered with the
<canvas> element. It's a very real use case though; not an average for hundreds or thousands of small tests.
The score for the HTML5-Benchmark takes the total time the browser spent rendering frames and a big penalty for pauses into account (the exact formula is
1000000/(sqrt(totalTime) + lagTime * 0.1)). The benchmark automatically runs at a “reasonable” screen size - i.e. a screen size that makes sense for a game on the particular device. Thus, the final score is a real indicator for the browser's ability to smoothly run HTML5 games.
A score of about 1500 means the game is playable. Note that the benchmark actually runs a bit slower on mobile devices than the game itself. This is because the game utilizes a special pre-rendered background mode on mobile devices, but doesn't do so in the benchmark to keep the scores comparable.
It's almost comical how bad the Android's Browser performed in my tests: the Samsung Galaxy Tab 10.1 is actually slower than a four year old 1st gen iPod Touch. Firefox 4 on the same tablet performs a lot better though.
The desktop version of Firefox 6 still suffers a lot from the occasional GC pauses. This makes Firefox the last choice if you want to enjoy HTML5 games. The performance decrease from the stable Chrome 13 to the dev version of Chrome 15 is quite strange too. It may have to do something with the new hardware acceleration for Canvas' 2d context.
There are some things that the benchmark can't measure. Opera for instance has a very steady timing and never misses a frame. On my machine however, the benchmark still looks choppy - probably because the benchmark's timer isn't synchronized with the refresh rate of the display. Opera essentially renders frames that it never displays. The new requestAnimationFrame() could fix this, but it hasn't arrived in all browsers yet.
This leads me to ask a few questions: shouldn't browser vendors concentrate on fixing
setInterval() first, before implementing
requestAnimationFrame()? And shouldn't it behave more like
setInterval() instead of
setTimeout()? That is – I don't want to call a function to schedule each and every frame I want to draw, but just call it once and let the browser schedule the frames.
requestAnimationFrame() looks good on paper, but it doesn't actually produce visually better results than
setInterval() in those browsers that implement it.
On my machine, Chrome 13 and Firefox 6 awkwardly jump between 120 and 60 FPS when using it (it's disabled in the benchmark) and even without GC pauses there are noticeable hiccups, while IE9 is able to produce perfectly smooth animations without it. At the moment
requestAnimationFrame() is truly worthless for games.
So, no – we aren't fast yet. But we're getting there.