Dealing with Premultiplied alpha on iOS

In computer graphics there are two different ways to represent the opacity of a color value: Straight alpha and Premultiplied alpha.

When using straight, also known as linear, alpha:
- RGB values specify the color of the thing being drawn
- The alpha value specifies how solid it is

In this world, RGB and alpha are independent. We can change one without affecting the other. To make an object fade out, we would gradually reduce its alpha value while leaving RGB unchanged.

When using premultiplied:
- RGB specifies how much color the thing being drawn contributes to the output
- The alpha value specifies how much it obscures whatever is behind it.

In this world, RGB and alpha are linked. To make an object transparent we must reduce both its RGB (to contribute less color) and also its alpha (to obscure less of whatever is behind it). Fully transparent objects no longer have any color at all, so there is only one value that represents 100% transparency: RGB and alpha all zero.

Some days ago, I’d been racking my brains all day to solve one problem during an image processing on one of the apps on my work.

It's just a simple bit manipulation inside an image, and when I run the app on simulator on my machine, everything was fine, but on a real device I got different values.

Real device (iPhone XR in my case) renders using premultiplied and the simulator uses straight.

For example, for the same pixel I got this:

On Simulator: R = 254, G = 254, B = 254, A = 254
On a Real Device: R = 253, G = 253, B = 253, A = 254

Just refreshing a little bit about images and pixels:

In digital imaging, a pixel, is a physical point in a raster image, or the smallest addressable element in an all points addressable display device; so it is the smallest controllable element of a picture represented on the screen.

The RGB color model is an additive color model in which red, green, and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue. (Wikipedia)

How to solve? It's not hard, but when it's your first time dealing with could be.

First thing, my main problem was get the correct color of a pixel. So, I created this extension to help me:

extension UIImage {
    subscript (x: Int, y: Int) -> UIColor? {
        guard x >= 0 && x < Int(size.width) && y >= 0 && y < Int(size.height),
            let cgImage = cgImage,
            let provider = cgImage.dataProvider,
            let providerData = provider.data,
            let data = CFDataGetBytePtr(providerData) else {
            return nil
        }

        let numberOfComponents = 4
        let pixelData = ((Int(size.width) * y) + x) * numberOfComponents

        let r = CGFloat(data[pixelData]) / 255.0
        let g = CGFloat(data[pixelData + 1]) / 255.0
        let b = CGFloat(data[pixelData + 2]) / 255.0
        let a = CGFloat(data[pixelData + 3]) / 255.0

        return UIColor(red: r, green: g, blue: b, alpha: a)
    }
}

PS: Why (x: Int, y: Int) instead of a CGPoint? Answer: No problem.

Ok, now I have the UIColor for one exact point. But it's not considering premultiplied alpha.

Since Apple provides on CGImage a property called CGImageAlphaInfo who gives you which kind of alpha you have.

https://developer.apple.com/documentation/coregraphics/cgimagealphainfo?language=swift

And for premultiplied we have: (last and first) in my case last

https://developer.apple.com/documentation/coregraphics/cgimagealphainfo/kcgimagealphapremultipliedfirst?language=swift

kCGImageAlphaPremultipliedLast
The alpha component is stored in the least significant bits of each pixel and the color components have already been multiplied by this alpha value. For example, premultiplied RGBA.

That's it, surprisely easy to fix, the RGB values are multiplied by the alpha value, so, we just need to revert this.

premultiplied.R = (straight.R * straight.A / 255);
premultiplied.G = (straight.G * straight.A / 255);
premultiplied.B = (straight.B * straight.A / 255);
premultiplied.A = straight.A;

And we just need to do the opposite:

straight.R = (premultiplied.R / straight.A) * 255;
straight.G = (premultiplied.G / straight.A) * 255;
straight.B = (premultiplied.B / straight.A) * 255;
straight.A = premultiplied.A;

After figure out that, I create this extension to get the color of a pixel passing the CGImageAlphaInfo as parameter.

extension UIColor {
    func rgb(alphaInfo: CGImageAlphaInfo) -> (red: UInt8, 
                                            green: UInt8, 
                                             blue: UInt8, 
                                            alpha: UInt8) {

        var fRed: CGFloat = 0
        var fGreen: CGFloat = 0
        var fBlue: CGFloat = 0
        var fAlpha: CGFloat = 0

        if self.getRed(&fRed, green: &fGreen, blue: &fBlue, alpha: &fAlpha) {

            var iRed = fRed * 255.0
            var iGreen = fGreen * 255.0
            var iBlue = fBlue * 255.0
            let iAlpha = fAlpha * 255.0

            switch alphaInfo {
            case .premultipliedLast:
                    iRed = (iRed / iAlpha) * 255
                    iGreen = (iGreen / iAlpha) * 255
                    iBlue = (iBlue / iAlpha) * 255
            default:
                break
            }

            return (red:UInt8(iRed.rounded()),   

                  green:UInt8(iGreen.rounded()),   

                   blue:UInt8(iBlue.rounded()), 

                  alpha:UInt8(iAlpha.rounded()))
        } else {
            // Could not extract RGBA components:
            return (0, 0, 0 , 0)
        }
    }
}

Fixed! Now it's working. After that is time to refactor these extensions to something better. For this post I end here. Thanks for reading.