PHOBOSLAB

Blog Home

Drawing Pixels is Hard

Way harder than it should be.

Back in 2009 when I first started to work on what would become my HTML5 game engine Impact, I was immediately presented with the challenge of scaling the game screen while maintaining crisp, clean pixels. This sounds like an easy problem to solve – after all Flash did this from day one and "retro" games are a big chunk of the market, especially for browser games, so it really should be supported – but it's not.

Let's say I have a game with an internal resolution of 320×240 and I want to scale it up 2x to 640×480 when presented on a website. With the HTML5 Canvas element, there are essentially two different ways to do this.

a) Creating the Canvas element in the scaled up resolution (640×480) and draw all images at twice the size:

var canvas = document.createElement('canvas');
canvas.width = 640;
canvas.width = 480;

var ctx = canvas.getContext('2d');
ctx.scale( 2, 2 );
ctx.drawImage( img, 0, 0 );

b) Using CSS to scale the Canvas – In my opinion this is the cleaner way to do it. It nicely decouples the internal canvas size from the size at which it is presented:

var canvas = document.createElement('canvas');
canvas.width = 320;
canvas.width = 240;
canvas.style.width = '640px';
canvas.style.width = '480px';

var ctx = canvas.getContext('2d');
ctx.drawImage( img, 0, 0 );

Both methods have a problem though – they use a bilinear (blurry) filtering instead of nearest-neighbor (pixel repetition) when scaling.

What I wanted (left) vs. what I got (right)

For the internal scaling approach (method a), you can set the context's imageSmoothingEnabled property to false in order to have crisp, nearest-neighbor scaling. This has been supported in Firefox for a few years now, but Chrome only just recently implemented it and it is currently unsupported in Safari (including Mobile Safari) and Internet Explorer (test case).

When doing the scaling in CSS (method b), you can use the image-rendering CSS property to specify the scaling algorithm the browser should use. This works well in Firefox and Safari, but all other browsers simply ignore it for the Canvas element (test case).

Of course Internet Explorer is the only browser that currently doesn't support any of these methods.

Not having crisp scaling really bothered me when I initially started to work on Impact. Keep in mind that at the time no browser supported either of the two methods described above. So I experiment a lot to find a solution.

And I found one. It's incredibly backwards and really quite sad: I do the scaling in JavaScript. Load the pixel data of each image, loop through all pixels and copy and scale the image, pixel by pixel, into a larger canvas then throw away the original image and use this larger canvas as the source for drawing instead.

var resize = function( img, scale ) {
    // Takes an image and a scaling factor and returns the scaled image
    
    // The original image is drawn into an offscreen canvas of the same size
    // and copied, pixel by pixel into another offscreen canvas with the 
    // new size.
    
    var widthScaled = img.width * scale;
    var heightScaled = img.height * scale;
    
    var orig = document.createElement('canvas');
    orig.width = img.width;
    orig.height = img.height;
    var origCtx = orig.getContext('2d');
    origCtx.drawImage(img, 0, 0);
    var origPixels = origCtx.getImageData(0, 0, img.width, img.height);
    
    var scaled = document.createElement('canvas');
    scaled.width = widthScaled;
    scaled.height = heightScaled;
    var scaledCtx = scaled.getContext('2d');
    var scaledPixels = scaledCtx.getImageData( 0, 0, widthScaled, heightScaled );
    
    for( var y = 0; y < heightScaled; y++ ) {
        for( var x = 0; x < widthScaled; x++ ) {
            var index = (Math.floor(y / scale) * img.width + Math.floor(x / scale)) * 4;
            var indexScaled = (y * widthScaled + x) * 4;
            scaledPixels.data[ indexScaled ] = origPixels.data[ index ];
            scaledPixels.data[ indexScaled+1 ] = origPixels.data[ index+1 ];
            scaledPixels.data[ indexScaled+2 ] = origPixels.data[ index+2 ];
            scaledPixels.data[ indexScaled+3 ] = origPixels.data[ index+3 ];
        }
    }
    scaledCtx.putImageData( scaledPixels, 0, 0 );
    return scaled;
}

This worked surprisingly well and has been the easiest way to scale up pixel-style games in Impact from day one. The scaling is only done once when the game first loads, so the performance hit isn't that bad, but you still notice the longer load times on mobile devices or when loading big images. After all, it's a stupidly costly operation do to, even in native code. We usually use GPUs for stuff like that.

All in all, doing the scaling in JavaScript is not the "right" solution, but the one that works for all browsers.

Or rather worked for all browsers.

Meet the retina iPhone

When Apple introduced the iPhone 4, it was the first device with a retina display. The pixels on the screen are so small, that you can't discern them. This also means, that in order to read anything on a website at all, this website has to be scaled up 2x.

So Apple introduced the devicePixelRatio. It's the ratio of real hardware pixels to CSS pixels. The iPhone 4 has a device pixel ratio of 2, i.e. one CSS pixel is displayed with 2 hardware pixels on the screen.

This also means that the following canvas element will be automatically scaled up to 640×480 hardware pixels on a retina device, when drawn on a website. Its internal resolution, however, still is 320×240.

<canvas width="320" height="240">

This automatic scaling again happens with the bilinear (blurry) filtering by default.

So, in order to draw at the native hardware resolution, you'd have to do your image scaling in JavaScript as usual but with twice the scaling factor, create the canvas with twice the internal size and then scale it down again using CSS.

Or, in recent Safari's, use the image-rendering: -webkit-optimize-contrast; CSS property. Nice!

This certainly makes things a bit more complicated, but devicePixelRatio was a sane idea. It makes sense.

Meet the retina MacBook Pro

For the new retina MacBook Pro (MBP), Apple had another idea. Instead of behaving in the same way as Mobile Safari on the iPhone, Safari for the retina MBP will automatically create a canvas element with twice the internal resolution than you requested. In theory, this is quite nice if you only want to draw shapes onto your canvas - they will automatically be in retina resolution. However, it significantly breaks drawing images.

Consider this Canvas element:

<canvas width="320" height="240"></canvas>

On the retina MBP, this will actually create a Canvas element with an internal resolution of 640×480. It will still behave as if it had an internal resolution of 320×240, though. Sort of.

This ingenious idea is called backingStorePixelRatio and, you guessed it, for the retina MBP it is 2. It's still 1 for the retina iPhone. Because… yeah…

(Paul Lewis recently wrote a nice article about High DPI Canvas Drawing, including a handy function that mediates between the retina iPhone and MBP and always draws in the native resolution)

Ok, so what happens if you now draw a 320×240 image to this 320×240 Canvas that in reality is a 640×480 Canvas? Yep, the image will get scaled using bilinear (blurry) filtering. Granted, if it wouldn't use bilinear filtering, this whole pixel ratio dance wouldn't make much sense. The problem is, there's no opt-out.

Let's say I want to analyze the colors of an image. I'd normally just draw the image to a canvas element retrieve an array of pixels from the canvas and then do whatever I want to do with them. Like this:

ctx.drawImage( img, 0, 0 );
var pixels = ctx.getImageData( 0, 0, img.width, img.height );
// do something with pixels.data...

On the retina MBP you can't do that anymore. The pixels that getImageData() returns are interpolated pixels, not the original pixels of the image. The image you have drawn to the canvas was first scaled up, to meet the bigger backing store and then scaled down again when retrieved through getImageData(), because getImageData() still acts as if the canvas was 320×240.

Fortunately, Apple also introduced a new getImageDataHD() method to retrieve the real pixel data from the backing store. So all you'd have to do is draw your image to the canvas with half the size, in order to draw it at the real size. Confused yet?

var ratio = ctx.webkitBackingStorePixelRatio || 1;
ctx.drawImage( img, 0, 0, img.width/ratio, img.height/ratio );

var pixels = null;
if( ratio != 1 ) {
    pixels = ctx.webkitGetImageDataHD( 0, 0, img.width, img.height );
}
else {
    pixels = ctx.getImageData( 0, 0, img.width, img.height );
}

(Did I say it's called getImageDataHD()? I lied. You gotta love those vendor prefixes. Imagine how nice it would be if there also was a moz, ms, o and a plain variant!)

The "Good" News

Ok, take a deep breath, there are only 3 different paths you have to consider when drawing sharp pixels on a scaled canvas.


The CSS image-rendering property and the Canvas' imageSmoothingEnabled really make things a bit easier, but it would be nice if they were universally supported. Especially Safari is in desperate need for imageSmoothingEnabled-support, with all the crazy retina stuff they have going on.

Let me also go on record saying that backingStorePixelRatio was a bad idea. It would have been a nice opt-in feature, but it's not a good default. A comment from Jake Archibald on Paul Lewis' article tells us why:

<canvas> 2D is a bitmap API, it's pixel dependent. An api that lets you query individual pixels shouldn't be creating pixels you don't ask for.

Apple's backingStorePixelRatio completely breaks the font rendering in Impact, makes games look blurry and breaks a whole bunch of other apps that use direct pixel manipulation. But at least Apple didn't have to update all their dashboard widgets for retina resolution. How convenient!

Update September 18th 2012: To demonstrate the bug in Safari, I build another test case and filed a report with Apple.

Friday, September 14th 2012

25 Comments:

#1Trevor Norris – Friday, September 14th 2012, 01:57

Yeah... I love the idea of developing games using HTML5 and stuff, but it's posts like these that bring me screeching back to reality.

#2Michal Wroblewski – Friday, September 14th 2012, 02:03

Ha, HTML5 - Standard, let's do it our way.

#3 – fredo – Friday, September 14th 2012, 02:40

It seems like whenever Apple releases a new device we have to rebuild the web to work on it.

#4Mr Speaker – Friday, September 14th 2012, 13:04

Fantastic post - I've been pulling my hair out over my non-square pixels... thanks for the write up!

#5 – Adam Żochowski – Friday, September 14th 2012, 13:45

When scaling pixel art, wouldn't you want to use pixel scaling algorithms?

en.wikipedia.org/wiki/Pixel_art_scaling_algorithms

#6Dominic – Friday, September 14th 2012, 14:32

@Adam: I ported the HQX scaling algorithm to JavaScript a while ago, but (surprise) it's currently broken in Safari on retina MacBooks:
www.phoboslab.org/log/2010/12/hqx-scaling-in-javascript

Such algorithms are certainly nice to have, but I also really like big chunky pixels :)

#7 – Lorenza – Friday, September 14th 2012, 16:51

Thanks for researching all this pain and putting it in one place. The quicker we figure it out, the quicker we can toss it into an impedance-matching library somewhere to keep our eyes from bleeding.

I recognize that Apple doesn't want to ship a product that suddenly makes existing web content all tiny on their devices, but I there's no excuse for returning interpolated pixels from

getImageData
on a MBP Retina. Likewise, making the APIs different between two retina display devices from the same manufacturer (!!!) is a cruel, punishing blunder.

#8 – Lorenza – Friday, September 14th 2012, 16:52

Heh, I should've known better than to toss a code tag inline like GitHub backticks. Oops.

#9John Evans – Friday, September 14th 2012, 17:24

Wow. Just...wow.

#10 – Victor – Friday, September 14th 2012, 20:21

Dude, what a waste of time. Is this fun?

#11 – Jarod – Saturday, September 15th 2012, 00:17

Thanks a bunch for posting this. I've gone through similar struggles and it's insane how difficult it is to scale pixel art.

#12Nic – Saturday, September 15th 2012, 17:35

At first I wanted to suggest a simple optimization in the resize code that should really reduce the time to resize the picture. In fact I was really surprise that it was not there in the first place.

For each scaled pixel these two lines are called:

            var index = (Math.floor(y / scale) * img.width + Math.floor(x / scale)) * 4;
            var indexScaled = (y * widthScaled + x) * 4;


instead both variables could be simply be incremented. It's less math intensive, less memory access. So in theory this is a no brainer.

But before suggesting it I wanted to check by myself how much improvement it would be and also how javascript engines optimize the code.
So a quick test (results in the console): www.mx981.com/stuff/resize_bench/

Oh boy what a surprise when I saw that my attempt to optimize the code made things worse most of the time. I tried different things and it's nowhere close to the performance of the original function.
In opera my function is 6 times faster but in Safari it's 5 times slower and in chrome the original function perform way better.
I assume this is how javascript engines behave and that they get better when some coding style is used. But the disparity of result on some such small change is freaky and counter-intuitive.

@Dominic, can you share some of your experience in javascript code optimization and what is your overall best practices?

#13Dominic – Saturday, September 15th 2012, 20:44

@Nic: interesting benchmark for sure!

I usually don't care about such low level optimizations at all, but I'm really surprised that it makes such a huge difference here. I assumed that the array access is far more costly than the calculation of the index.

I played around a bit with your code and honestly - I have no idea what's going on. Somehow, the 'indexFloat+=' operation seems to be the culprit?!

#14Nic – Saturday, September 15th 2012, 23:41

It's not just that.
the second method I tested also recalculate the indexes each time but with a less complex calculation since there is only one loop. But this is still slower in some browser.

Traditionally accessing an array shouldn't be so heavy especially when it's linear but getting the pixel data (getImageData) is usually more heavy since the pixel data would probably need to be formatted. And of course multiple operation especially division are costly.
But with such result I wonder if this traditional view is still relevant.

I will try to dig a bit more into it tomorrow.

#15 – Neb – Tuesday, September 18th 2012, 15:41

Did you think about using SVGs and scaling?

Or would this mean a massive decline in performance?

#16Mathieu 'p01' Henri – Wednesday, September 19th 2012, 11:39

Adding the following method to @Nic's test

var index = 0;
var ref_indexScaled = 0
var ref_step=1/scale;
for( var y = 0; y < heightScaled; y++ ) {
  for( var x = 0; x < widthScaled; x++ ) {
    var i= index<<2;
    scaledPixels.data[ ref_indexScaled++ ] = origPixels.data[ i++ ];
    scaledPixels.data[ ref_indexScaled++ ] = origPixels.data[ i++ ];
    scaledPixels.data[ ref_indexScaled++ ] = origPixels.data[ i++ ];
    scaledPixels.data[ ref_indexScaled++ ] = origPixels.data[ i++ ];

    index+= ref_step;
  }
}


Yielded the following results in Opera Next:

Original - 2311ms
refactor - 112ms
hybrid - 2371ms
THE METHOD ABOVE - 3ms

As silly as it sounds, the Math.whatever() calls can be tricky to optimize/inline for JS engines. Whenever possible, prefer an arithmetic alternative ( use operators and co. instead of method calls ).

Hope that helps,
Cheers,

#17Amadeus – Friday, September 21st 2012, 08:18

@Nic

A couple things to keep in mind, if you cache the img.width and img.height into local vars you can also significantly increase the time required to perform some of the calculations.

This enters the world of micro-ops, but it can make a huge difference.

For example, the - only index calculation - I was able to reduce from 104ms to 61ms, that's a whole order of magnitude.

In other words, anytime you can remove look ups on the dom props, you can get big wins

#18 – Jackson – Tuesday, September 25th 2012, 02:25

Dom, so what does this mean for Impact and iOSImpact games requiring retina graphics? Is it still possible on Mobile Safari?

#19Dominic – Sunday, September 30th 2012, 02:33

@Jackson Mobile Safari is unaffected by this. Impact still works fine on all iPhones/iPads.

#20panzi – Wednesday, October 31st 2012, 23:40

Does using webgl (where supported) change anything?

#21panzi – Tuesday, January 8th 2013, 15:08

The new Firefox 18 supports retina and window.devicePixelRatio. Does this make yet another workaround necessary?

#22namuol – Friday, January 11th 2013, 09:03

Sad truths, but a great article on a topic I've been frustrated with for a solid year, now.

I just realized that you must've stumbled upon my StackOverflow question wherein talked about this particular problem, since you're using the test fiddle I made for it. (stackoverflow.com/questions/7615009)

Even funnier, I ended up going back to that question multiple times to update it; once I even posted a link to this article, since it was the only comprehensive source on the subject -- and this was _before_ I noticed you used the same fiddle!

It's a small world for us HTML5 engine developers. ;)

Cheers,
-lou

#23Rob Boerman – Sunday, March 17th 2013, 10:52

Great explanations of all the different scaling issues. It's good to have all that fragmented information in one well worked out post. Thanks for sharing!

#24Guilherme Iago – Thursday, July 11th 2013, 04:57

I could idolize you, but you're amazing, his knowledge .. I hope one day to have that level of knowledge you have .. or at least half ..

#25Diogo Schneider – Thursday, July 18th 2013, 02:31

I've stumbled upon this issue when I added fullscreen support for Starship, but only recently I filled a proper bug report for this on Chromium:
code.google.com/p/chromium/issues/detail?id=260739

It took ages for them to provide a working fix for their audio issue (code.google.com/p/chromium/issues/detail?id=107933), so please help me upvote that.

Thank you!

Post a Comment:

Comment: (Required)

(use <code> tags for preformatted text; URLs are recognized automatically)

Name: (Required)

URL:

Please type phoboslab into the following input field or enable Javascript. This is an anti-spam measure. Sorry for the inconvenience.