Snapchat Filters: How Do They Work?

About five times a day, I, along with 180 million daily users, open Snapchat to share small moments of my life with family and friends. (Truth be told, I almost always send videos of my dogs.)

A few times a week, I browse through Snapchat’s lenses (what we know as Snapchat filters) to see if there’s anything new.

I’m a big fan of Snapchat, not only because I’ve limited my Facebook and Instagram usage, but also because I grew up as an admirer of photo technology. Snapchat’s ability to entertain by swapping faces with a friend or making animated rainbows fall out of your mouth built on that longtime admiration and intrigued me.

I started using Snapchat in 2016, I’ve wondered how the augmented reality technology behind their filters worked. Since I needed an idea for my Fullstack Academy stackathon project, it was my opportunity to dive in deep to understand what was going on behind the curtain.
The technology came from Looskery, a Ukranian tech company acquired by Snapchat in 2015.

Before diving into Snapchat’s technology, let’s take a 30,000-foot view of the universe that is Computer Vision — which is how computers receive data from your camera to identify objects.

What is Computer Vision?

Computer vision is an interdisciplinary field that addresses how computers translate the large amount of data[deals with how computers can be made for gaining high-level understanding] from digital images or videos. From the perspective of engineering, it seeks to automate tasks that the human visual system typically does.

Computer vision is the reason that:

• Facebook can tag your friends in photos;
• you can deposit a check with your phone;
• self-driving cars can avoid hitting other cars or trees; and
• you can send snaps of your face with a dog nose.

How can computers see faces?

When a computer looks at an image, it only sees 1s and 0s.

All human faces share similar properties:

• the nose bridge is lighter than its sides
• the eye region is darker than the upper-cheeks
• the eye sockets are darker than the forehead

Using the Viola-Jones algorithm, computers scan through the image data to detect contrasts between light and dark pixel areas. When finding enough matches for facial features in the pixel data in one particular part of an image, the computer can identify a face in the photo.

The algorithm works well with frontal facing, but is unable to detect side faces.

How does Snapchat put a dog nose on my face?

Now let’s zoom back to learn how Snapchat uses that algorithm to map its filters to your face.

For a computer to put a dog nose on your face, it needs to do more complex calculations to locate your facial features. This is done by using an active shape model. Computers relying on this model draw on manual markings by humans that indicate the borders and facial features on thousands and thousands of images.

The data from these faces are then applied to an image from your phone’s camera, scaling it to the areas where it has already detected your face.

The model isn’t perfect, but it has a template for what the top of your mouth looks like. It looks for the pattern in your image and adjusts accordingly to match it.

It also takes into account the locations of other points on your face to create a 3D mask, which can be used to scale, rotate, and move as the data from your camera comes in frame by frame.

This 3D mask can then be re-formed to adapt to your facial features as they move (e.g., eyes or chin), add accessories, and set rainbows to come alive when you open your mouth.

Now you’re an augmented reality expert.

So now you know how Snapchat works, you can adopt this topic as your go-to cocktail party conversation starter or impress your significant other’s parents at the next family dinner.

I hope this was interesting and enjoyed reading it as much as I enjoyed doing the research. Happy snapping.