Way harder than it should be.
Back in 2009 when I first started to work on what would become my HTML5 game engine Impact, I was immediately presented with the challenge of scaling the game screen while maintaining crisp, clean pixels. This sounds like an easy problem to solve – after all Flash did this from day one and "retro" games are a big chunk of the market, especially for browser games, so it really should be supported – but it's not.
Let's say I have a game with an internal resolution of 320×240 and I want to scale it up 2x to 640×480 when presented on a website. With the HTML5 Canvas element, there are essentially two different ways to do this.
a) Creating the Canvas element in the scaled up resolution (640×480) and draw all images at twice the size:
var canvas = document.createElement('canvas');
canvas.width = 640;
canvas.width = 480;
var ctx = canvas.getContext('2d');
ctx.scale( 2, 2 );
ctx.drawImage( img, 0, 0 );
b) Using CSS to scale the Canvas – In my opinion this is the cleaner way to do it. It nicely decouples the internal canvas size from the size at which it is presented:
var canvas = document.createElement('canvas');
canvas.width = 320;
canvas.width = 240;
canvas.style.width = '640px';
canvas.style.width = '480px';
var ctx = canvas.getContext('2d');
ctx.drawImage( img, 0, 0 );
Both methods have a problem though – they use a bilinear (blurry) filtering instead of nearest-neighbor (pixel repetition) when scaling.
What I wanted (left) vs. what I got (right)
For the internal scaling approach (method a), you can set the context's imageSmoothingEnabled property to false in order to have crisp, nearest-neighbor scaling. This has been supported in Firefox for a few years now, but Chrome only just recently implemented it and it is currently unsupported in Safari (including Mobile Safari) and Internet Explorer (test case).
When doing the scaling in CSS (method b), you can use the image-rendering CSS property to specify the scaling algorithm the browser should use. This works well in Firefox and Safari, but all other browsers simply ignore it for the Canvas element (test case).
Of course Internet Explorer is the only browser that currently doesn't support any of these methods.
Not having crisp scaling really bothered me when I initially started to work on Impact. Keep in mind that at the time no browser supported either of the two methods described above. So I experiment a lot to find a solution.
And I found one. It's incredibly backwards and really quite sad: I do the scaling in JavaScript. Load the pixel data of each image, loop through all pixels and copy and scale the image, pixel by pixel, into a larger canvas then throw away the original image and use this larger canvas as the source for drawing instead.
var resize = function( img, scale ) {
// Takes an image and a scaling factor and returns the scaled image
// The original image is drawn into an offscreen canvas of the same size
// and copied, pixel by pixel into another offscreen canvas with the
// new size.
var widthScaled = img.width * scale;
var heightScaled = img.height * scale;
var orig = document.createElement('canvas');
orig.width = img.width;
orig.height = img.height;
var origCtx = orig.getContext('2d');
origCtx.drawImage(img, 0, 0);
var origPixels = origCtx.getImageData(0, 0, img.width, img.height);
var scaled = document.createElement('canvas');
scaled.width = widthScaled;
scaled.height = heightScaled;
var scaledCtx = scaled.getContext('2d');
var scaledPixels = scaledCtx.getImageData( 0, 0, widthScaled, heightScaled );
for( var y = 0; y < heightScaled; y++ ) {
for( var x = 0; x < widthScaled; x++ ) {
var index = (Math.floor(y / scale) * img.width + Math.floor(x / scale)) * 4;
var indexScaled = (y * widthScaled + x) * 4;
scaledPixels.data[ indexScaled ] = origPixels.data[ index ];
scaledPixels.data[ indexScaled+1 ] = origPixels.data[ index+1 ];
scaledPixels.data[ indexScaled+2 ] = origPixels.data[ index+2 ];
scaledPixels.data[ indexScaled+3 ] = origPixels.data[ index+3 ];
}
}
scaledCtx.putImageData( scaledPixels, 0, 0 );
return scaled;
}
This worked surprisingly well and has been the easiest way to scale up pixel-style games in Impact from day one. The scaling is only done once when the game first loads, so the performance hit isn't that bad, but you still notice the longer load times on mobile devices or when loading big images. After all, it's a stupidly costly operation do to, even in native code. We usually use GPUs for stuff like that.
All in all, doing the scaling in JavaScript is not the "right" solution, but the one that works for all browsers.
Or rather worked for all browsers.
Meet the retina iPhone
When Apple introduced the iPhone 4, it was the first device with a retina display. The pixels on the screen are so small, that you can't discern them. This also means, that in order to read anything on a website at all, this website has to be scaled up 2x.
So Apple introduced the devicePixelRatio. It's the ratio of real hardware pixels to CSS pixels. The iPhone 4 has a device pixel ratio of 2, i.e. one CSS pixel is displayed with 2 hardware pixels on the screen.
This also means that the following canvas element will be automatically scaled up to 640×480 hardware pixels on a retina device, when drawn on a website. Its internal resolution, however, still is 320×240.
<canvas width="320" height="240">
This automatic scaling again happens with the bilinear (blurry) filtering by default.
So, in order to draw at the native hardware resolution, you'd have to do your image scaling in JavaScript as usual but with twice the scaling factor, create the canvas with twice the internal size and then scale it down again using CSS.
Or, in recent Safari's, use the image-rendering: -webkit-optimize-contrast; CSS property. Nice!
This certainly makes things a bit more complicated, but devicePixelRatio was a sane idea. It makes sense.
Meet the retina MacBook Pro
For the new retina MacBook Pro (MBP), Apple had another idea. Instead of behaving in the same way as Mobile Safari on the iPhone, Safari for the retina MBP will automatically create a canvas element with twice the internal resolution than you requested. In theory, this is quite nice if you only want to draw shapes onto your canvas - they will automatically be in retina resolution. However, it significantly breaks drawing images.
Consider this Canvas element:
<canvas width="320" height="240"></canvas>
On the retina MBP, this will actually create a Canvas element with an internal resolution of 640×480. It will still behave as if it had an internal resolution of 320×240, though. Sort of.
This ingenious idea is called backingStorePixelRatio and, you guessed it, for the retina MBP it is 2. It's still 1 for the retina iPhone. Because… yeah…
(Paul Lewis recently wrote a nice article about High DPI Canvas Drawing, including a handy function that mediates between the retina iPhone and MBP and always draws in the native resolution)
Ok, so what happens if you now draw a 320×240 image to this 320×240 Canvas that in reality is a 640×480 Canvas? Yep, the image will get scaled using bilinear (blurry) filtering. Granted, if it wouldn't use bilinear filtering, this whole pixel ratio dance wouldn't make much sense. The problem is, there's no opt-out.
Let's say I want to analyze the colors of an image. I'd normally just draw the image to a canvas element retrieve an array of pixels from the canvas and then do whatever I want to do with them. Like this:
ctx.drawImage( img, 0, 0 );
var pixels = ctx.getImageData( 0, 0, img.width, img.height );
// do something with pixels.data...
On the retina MBP you can't do that anymore. The pixels that getImageData() returns are interpolated pixels, not the original pixels of the image. The image you have drawn to the canvas was first scaled up, to meet the bigger backing store and then scaled down again when retrieved through getImageData(), because getImageData() still acts as if the canvas was 320×240.
Fortunately, Apple also introduced a new getImageDataHD() method to retrieve the real pixel data from the backing store. So all you'd have to do is draw your image to the canvas with half the size, in order to draw it at the real size. Confused yet?
var ratio = ctx.webkitBackingStorePixelRatio || 1;
ctx.drawImage( img, 0, 0, img.width/ratio, img.height/ratio );
var pixels = null;
if( ratio != 1 ) {
pixels = ctx.webkitGetImageDataHD( 0, 0, img.width, img.height );
}
else {
pixels = ctx.getImageData( 0, 0, img.width, img.height );
}
(Did I say it's called getImageDataHD()? I lied. You gotta love those vendor prefixes. Imagine how nice it would be if there also was a moz, ms, o and a plain variant!)
The "Good" News
Ok, take a deep breath, there are only 3 different paths you have to consider when drawing sharp pixels on a scaled canvas.
- Check the backingStorePixelRatio. If it's not
1, divide your canvas size and the destination size of all image draw calls by it, then scale the canvas element up using CSS and the image-rendering property. (Safari)
- Check if the
imageSmoothingEnabled property is available and if so, set it to false. Create your Canvas in the final, scaled size and draw all images with your scaling factor. Don't use CSS to scale the Canvas. (Chrome, Firefox)
- Use JavaScript to scale up all images at load time. (Internet Explorer)
The CSS image-rendering property and the Canvas' imageSmoothingEnabled really make things a bit easier, but it would be nice if they were universally supported. Especially Safari is in desperate need for imageSmoothingEnabled-support, with all the crazy retina stuff they have going on.
Let me also go on record saying that backingStorePixelRatio was a bad idea. It would have been a nice opt-in feature, but it's not a good default. A comment from Jake Archibald on Paul Lewis' article tells us why:
<canvas> 2D is a bitmap API, it's pixel dependent. An api that lets you query individual pixels shouldn't be creating pixels you don't ask for.
Apple's backingStorePixelRatio completely breaks the font rendering in Impact, makes games look blurry and breaks a whole bunch of other apps that use direct pixel manipulation. But at least Apple didn't have to update all their dashboard widgets for retina resolution. How convenient!
Update September 18th 2012: To demonstrate the bug in Safari, I build another test case and filed a report with Apple.
Less talk, more action – Apple just approved two of my JavaScript games for the AppStore: Biolab Disaster and Drop. Both are free; go check them out. You can also play them in your browser here and here.
Both games are pretty simple (the source for Drop is only 300 odd lines long) and written with my JavaScript Game Engine Impact.
These are certainly not the first games written in JavaScript to be available in the AppStore. Tools like AppMobi, PhoneGap or Titanium make it easy to bundle some HTML pages and JavaScript together in an App and display them in a UIWebView, which is basically just a browser window (correction: Titanium doesn't use a UIWebView but instead has some native bindings). Games written with Impact already work okay-ish in the iPhone's browser and thus also in AppMobi and PhoneGap.
So what's so special about these two games now? They don't use PhoneGap or Titanium. They don't even use a UIWebView. Instead, they bypass the iPhone's browser altogether and use Apple's JavaScript interpreter (JavaScriptCore) directly. All graphics are rendered with OpenGL instead of in a browser window and all sound and music is played back with OpenAL instead of… well, having no sound at all.
What makes this possible is a compatibility layer that mimics the HTML5 Canvas and Audio APIs but is implemented with OpenGL and OpenAL behind the scenes. Think of it as a browser that can only display a Canvas element and play Audio elements, but does not render generic HTML pages. A browser perfectly suited for HTML5 games.
This means you can take your JavaScript games written for Impact and run them on iOS with perfect sound and touch input and way better drawing performance than with Mobile Safari. Now, to be clear, I only implemented a bare minimum of the Canvas API - just enough to be able to run Impact. The whole thing is still in a very experimental state.
If you have a license for Impact you will find the complete source code for this all on your download page. I also wrote some basic documentation to get you started. In theory, you don't have to know anything about Objective-C to use this, but at this stage some Objective-C knowledge will sure come in handy. Again, this is very experimental. Don't expect it to work at all.
If you've been following this blog for a while, you might remember a post from October 2010 where I attempted the exact same thing. Back then I used the JavaScriptCore library that is already available on iOS and used by Apple in Mobile Safari. The problem was, that this library is "private", meaning that Apple does not want you to use it. And since you can't publish anything in the AppStore that uses private libraries I abandoned the idea.
However I recently revisited the project, because there was still a chance to make this work: JavaScriptCore is a part of the open source WebKit project. Instead of using the private library that comes with iOS, you theoretically could compile your own version of this library and bundle it together with your App. Which is exactly what I did.
Since Apple does not provide any project files to compile JavaScriptCore for iOS (presumably to annoy us) and JavaScriptCore itself uses some of iOS' private APIs, compiling this beast into a static library - in an App Store compatible fashion - took me a few days.
I also had to make a small sacrifice: JavaScriptCore uses libicu to sort strings according to a unicode locale. Sadly, libicu is also private on iOS and bundling it is not an option because of its size. So I got rid of libicu completely. This means that only ASCII strings are now sorted correctly (e.g. the umlaut "Ä" will come after "Z", not after "A" as it should). Other than that, the JavaScript library should behave exactly as the private one that comes with iOS.
Also, the JavaScriptCore library bundled with iOSImpact does not use the JIT compiler (Nitro). You can't allocate executable memory on the iPhone and if Apple doesn't lift that restriction, there's nothing I can do about it. However, Apple recently made an exception for Mobile Safari – if they would make their JavaScriptCore API public, they probably could enable the JIT for everyone. That's a big if though; I don't see it happening, because Apple loves native code (Objective-C) and hates scripting languages.
I was afraid Apple would reject the two games for some obscure reason, but they didn't. Which leaves me to wonder why the JavaScriptCore library that comes with iOS is private in the first place. Bundling JavaScriptCore with your App adds about 2MB in size – not much, but I fail to see how this can be in Apple's interest.
Anyway, the performance of both games is pretty good. I still get some occasional slowdowns (~20fps) on my iPhone3GS in Biolab Disaster when there are too many particles on the screen, but it remains playable at all times. It's also nice to have perfectly working sound on iOS now - something even some desktop browsers still struggle with.
(Silly title, I know, but how could I ever resist such an opportunity?)
My HTML5 Game Engine Impact is now ready. It took some time, but I think it was worth it. I'm proud of what I have achieved and I hope you'll like it too.
Part of why it took so long to put it all together is that it now runs on the iPhone, iPod Touch and iPad. Try it yourself at playbiolab.com and impactjs.com/drop or watch a short video:
All those platforms still have their problems with sound and the iPhone 4 has a hard time filling all its pixels, but the games remain to be playable even on the 1st gen iPod Touch. You can read a bit more about Impact on mobile platforms in the documentation.
Even with iOS support, it might come as a shock to some of you that I am selling Impact, rather than releasing it for free. I love free and open source software and I've been contributing stuff for quite some time now. I had a hard time thinking about whether to release my Game Engine for free. The reason I decided to charge for Impact is a) it is easily the biggest thing I've ever made and I'd love to continue working on it full time, and b) I believe it is worth the money.
Ironically, my decision to sell Impact set back the release date quite a bit. If I'm selling something, I want it to be worth every penny. And even though the engine hasn't been far from completion for some time, I hadn't written a single line of documentation.
I feel that a good documentation is crucial for the success and adoption of any software project. So I set myself the goal to write the best documentation I possibly could.
I'm not a big fan of inline documentation (with documentation generators like JSDoc) because it tends to clutter the source code with trivial statements and – more importantly – makes it easy to write bad documentation. If you are writing the documentation separately from the code, you think about it differently. You think about the documentation as something that works without the source code, something that makes sense without the source code.
You rarely see code examples in automatically generated documentations, but for me as a developer, code examples are oftentimes exactly what I need. Take a look at the documentation for the ig.Entity Class - one of the more complex classes of Impact. This is something documentation generators just can't do.
Of course it took me longer to write the documentation separately than it would have if I wrote it inline, but this is only because it is more in-depth, more thorough.
But don't take my word for it. Please see for yourself!
On a lighter note, I'm currently sending out a few thousand emails to those who signed up on the old Impact landing page. I'm using a 10-line PHP script for that. Let's see how this turns out...
Before I go into some details of my HTML5 game Biolab Disaster, let me use my 15 minutes of fame to say the following:
Safari, please get your shit together! You are a very nice browser; my game runs with an excellent frame rate and everything works fine. But please (please!) support the Ogg Vorbis codec for Audio elements. There is no reason not to. I had to encode all my sound files in Ogg Vorbis and MP3, just because of you, Safari. You make my life unnecessarily difficult.
(I could now go on and say the same thing about Microsoft's upcoming IE9, but I don't care enough about their half-assed attempt to build a browser)
Biolab Disaster was completely build for HTML5 – that is, JavaScript and the new Canvas and Audio elements. It doesn't need any Plugins, just a good browser. The game was initially conceived as a demo for my Impact Game Engine, but it became much more in the process. I made a short making-of video that shows how the game and the engine were built:
Despite this all being a relatively new technology, the browser support is pretty good already. When I began with this project it wouldn't run in Opera at all; now Opera has one of the best Canvas and Audio implementations around. Current versions of Chrome and Firefox still have some sound issues (lag) but otherwise run the game fine. I did some performance tests on different browsers and platforms:
Frames per Second
A frame rate below 30 starts to feel “chunky” and anything below 20 is not really playable – or at least no fun to play. The iPhone 3GS and iPad therefore sadly fall into the “unplayable” category. The performance on an iPhone 4 should be significantly better, but I don’t have access to one at the moment. Mobile Safari also won't load the sound files correctly as it seems – the game is stuck in the preloader. I will look into it a bit closer in the coming days.
It's worth noting that – regardless of the browser – JavaScript performance was never an issue. The real bottleneck is the Canvas API. All browsers spend about 90% of their time drawing into the Canvas. Especially Firefox is a bit on the slow side, but with the upcoming hardware acceleration in Firefox 4, performance shouldn't be a problem anymore.
Biolab Disaster formulates the playable truth that it makes sense to create games for HTML5. Not only is the result on par with current Flash games, but also is the development process incredibly smooth and satisfying. The hurdles that a new technology such as HTML5 sets, were already overcome by the game engine. For the development of the game itself, I didn’t have to deal with any browser or platform issues at all.
With Microsoft delivering some HTML5 support in Internet Explorer 9 and JavaScript and rendering performance steadily increasing, I can’t see any reason why JavaScript and Canvas would not be the gaming platform of the coming years that finally removes Flash from its quasi monopoly.
See for yourself.
