How Does Facial Recognition Work?

Facial recognition technology works by detecting a face in an image or video, then using a deep neural network to convert its features into a numerical "faceprint." The system compares that faceprint against a database of stored faces and reports the closest match. Modern systems learn these features automatically rather than measuring fixed distances by hand.

Facial recognition is an advanced technology that detects and identifies human faces from an image or video. A facial recognition system uses biometrics to map the unique features of a face, turns them into a string of numbers, and then compares that information against a large database of recorded faces to find a correct match.

Facial recognition is considered one of the leading methods of biometric recognition, which identifies people by measuring some aspect of their individual physiology or anatomy. It is also one of the fastest-growing biometric technologies: the global facial recognition market was worth roughly $5-6 billion in 2024 and is forecast to keep climbing through the rest of the decade. The reason is simple. Facial recognition has a wide range of commercial applications and, unlike fingerprint or iris scanning, it can work at a distance without anyone touching a sensor. It can be used for everything from surveillance to unlocking your phone to targeted marketing.

History Of Facial Recognition

The idea is older than you might think. Back in the 1960s, mathematician Woodrow Bledsoe built a system that classified faces by manually plotting the coordinates of features like the eyes, nose and mouth. The field really took off in the early 1990s, though, when the United States Department of Defense launched its DARPA-funded FERET program to build software that could automatically identify faces in surveillance footage. The Defense Department roped in eminent university scientists and experts in the field, providing them with research financing and a large database of facial images to train on.

Facial recognition made bold headlines in early 2001, right after it was used in a public space for the first time. At Super Bowl XXXV in Tampa, a system called Facefinder quietly scanned roughly 100,000 spectators against a police database, flagging 19 people with petty criminal records (no one was detained). Word only got out days later, and the press promptly dubbed it the "Snooper Bowl." Soon after, facial recognition systems were installed in other sensitive parts of the US to keep track of felonious activities.

Although facial recognition is the fastest-growing biometric technology, it also happens to be the most controversial. After the 9/11 tragedy, many people supported the use of this new technology, but as the technology made deeper inroads to our lives, many realized that it could pose a threat to individual privacy and could also potentially lead to identity theft. No matter which side of this debate you’re on, it is worth knowing how this fast-growing technology works and what it can do.

How Facial Recognition Works

A facial recognition setup consists of advanced cameras that capture photos of people who pose or simply walk by, and sophisticated software working on those pictures will attempt to find the right match from the vast database to identify the person(s) in the image. Now, let’s take a closer look at the technical details of how these systems work.

As mentioned earlier, facial recognition methods vary slightly, depending on the application and manufacturer, but they generally involve a series of steps that serve to capture, process, analyze and match the captured face to a database of recorded images. These basic steps are:

1. Detection:

When the facial recognition system is attached to a video surveillance system, the recognition software scans the camera’s field of view for anything that looks like a face. Upon detecting each face-like pattern, it sends that region to the system for further processing. The system then estimates the head’s position, orientation, and size. These systems work best when the face is roughly square-on to the camera; the more a face turns away from the lens, the more features get hidden and the harder it becomes to detect and identify reliably.

2. Normalization:

The captured face is scaled, rotated and aligned into a standard pose and size so that two photos of the same person line up neatly. This is called normalization. Early systems then measured the geometry of the face by hand, gauging factors such as the distance between the eyes, the thickness of the lips, and the distance between the chin and the forehead. Today, almost every serious system replaces that step with a deep convolutional neural network. Trained on millions of faces, the network learns on its own which subtle features matter, and it boils the entire face down into a compact list of numbers, often a vector of 128 or 512 values, known as a faceprint or facial signature.

3. Representation:

This string of numbers is the face’s unique code, and storing the face as numbers (rather than as a picture) is what makes the next step so fast. Two faceprints can be compared with simple arithmetic, so a single query can be checked against millions of stored faces in a fraction of a second.

4. Matching:

This is the final stage, in which the new faceprint is compared to the stored ones. The system measures the mathematical "distance" between the vectors. The closer two faceprints sit, the more likely they belong to the same person. If the best match falls within a set threshold, the software returns the details of the matched face and notifies the end user. The best modern algorithms are remarkably good at this, but they are not flawless. A landmark 2019 study by the US National Institute of Standards and Technology (NIST) found that many algorithms misidentified people of color, women, the elderly and children at higher rates than middle-aged white men, a bias that matters a great deal when the technology is used by police or at a border.

Applications Of Facial Recognition

National Security

A lot of organizations and businesses are using facial recognition, albeit for varying purposes. Governments across the globe are using facial recognition systems at airports to monitor people coming and going from their country. The US Department of Homeland Security, for instance, has a system to identify people who have overstayed their visas or may be under criminal investigation.

Apple iPhone

Apple’s iPhone first made facial recognition a household term. Since then, most mid-range to high-end smartphones come with a face unlock feature to authenticate the phone. Face unlock is a form of facial recognition that ensures that you are actually you when attempting to access your phone. Apple’s Face ID, introduced with the iPhone X in 2017, is arguably the most robust face unlock out there. Rather than relying on a flat 2D photo, its TrueDepth camera projects more than 30,000 invisible infrared dots onto your face to build a 3D depth map, which is far harder to fool with a printed photo. Apple puts the odds of a random person unlocking your iPhone at less than one in a million.

Facebook

For years, Facebook used a deep-learning algorithm called DeepFace to detect faces when you uploaded a photo, then suggested tagging the people it recognized. Its 2014 research reported about 97% accuracy on a standard benchmark, roughly matching how well humans do at the same task, a striking demonstration of how good the technology had become. In November 2021, however, Facebook’s parent company Meta shut the system down and deleted the faceprints of more than a billion users, citing growing privacy concerns and the lack of clear rules from regulators. It remains one of the largest reversals in the technology’s history.

Churches!

Interestingly, even some religious groups have started to use facial recognition to keep tabs on their congregations! Hundreds of churches have used a system called Churchix to scan their pews and record who shows up. The software lets church management track members and how regularly they attend, which (unsurprisingly) has raised its own share of privacy eyebrows.