Visualizing the OpenCV Detection Process

This video visualizes the detection process of OpenCV’s face detector. The algorithm utilizes the Viola Jones method of calculating the integral image and then performing some calculations on all the areas defined by the black and white rectangles. The sub-window (in red) is scanned across the image at various scales to detect if there is a potential face. In the post-processing stage all the potential faces are checked for overlaps. Typically, 2 or 3 overlapping rectangles are required to confirm a face. Loner rectangles are rejected as false-positives.
This visualization was done as part of the documentation for CV Dazzle, camouflage from face detection. For more information, visit http://cvdazzle.com

In this image the dotted lines represent all possible face detections, including false positives, before post-processing. Solid lines indicate a positive face detection. Each of the different colors represent the different Haar Cascade face profiles that were used. Images without any solid lines are classified as false-positives and have beaten the face detector.

Post-processing in OpenCV helps remove false positives. Though false positives are still possible, they are reduced by searching for overlapping detections. Only detections that overlap at least 3 (default) other rectangles are considered to be a face. The default setting can be changed to 2 for a higher detection rate, but with more false positives. Setting the overlap minimum to 0 returns all possible detections, as shown here by the dotted lines.

The solid lines represent the faces. They are confirmed by multiple overlapping rectangles. And the colors correspond to the difference detection profiles. OpenCV includes four robust profiles for detecting faces in a frontal view. These profiles include the “frontalface_default.xml”, “frontalface_alt.xml”, “frontalface_alt2.xml”, and “frontalface_alt_tree.xml” files. Each detector returns different noticeably different results.

OpenCV is a computer vision framework available in many different programming languages. For this test, I used the Java port created for Processing from http://ubaa.net/shared/processing/opencv/opencv_image.html and then confirmed the results using the standard framework in C.

The images on the bottom row without any solid lines indicate that no faces were found. Although some potential faces were found. They were ignored, because there was not enough confidence.

These images show a proof of concept hair + makeup pattern capable of deceiving the OpenCV face detector. It was developed as my master’s thesis project at the Interactive Telecommunication Program at NYU. A more detailed report will be posted on CVDazzle.com when it’s ready.

Model: Jen Jaffe, Hair: Pia Vivas, Makeup: Leigh Brown

CV Dazzle vs PhotoTagger from Adam Harvey on Vimeo.

CV Dazzle is camouflage from face detection. ItTweet

Test patterns based on reversing Haar cascade files.

In the test images above, the original face was overlaid with hand-made variations of the true Haar cascade points of interest (POI) and the reversed POI. Variations of original POI yielded nearly 100% positive results except for the all white overlay (top middle). It was surprising the rest of the all-black/white pixels still tested positive, so one missed face was not too disappointing. All of the expected negatives tested negative. (Images in red are positive faces. Unaltered images were negative)

The images with a light overlay have white pixels where dark pixels should be if there, statistically, was face. Images with a dark overlay have black pixels where the light pixels should be. Because they are all custom variations of the original patterns (top 2 rows), they don’t all hide the face. But the patterns that do hide the images are surprisingly small. Note the small white accents that yielded a negative result and the large black pixel-beard that tested positive.

Visualizing the features that Haar-based face detection programs use to find faces.

When you use face detection in Processing and OpenFrameworks you’re using an algorithm developed in 2001 called the Viola-Jones method. Though seemingly old, it’s still a robust and efficient way to detect faces in real time. It works great in Java, C++ and even Flash as you can see from these examples in Processing, OpenFrameworks, and Flash.

The algorithm looks at the image (or video) and tries to match the Haar features (in black and white) to features of the image. The more features match, the more likely it is that the image contains a face. As its name implies, a cascade is made of a series of feature-tests. For a face to be detected, the algorithm must match a cascade or series of features in the expected locations. The examples above and below show the Haar features from the cascade files that ship with OpenCV. For more information on how face detection works, I recommend starting with a 2007 article from Servo magazine about how face detection works.

To do: Find the order and weight of the trees in the cascade to determine which are the most and least important features. (NB: the image of Lena is popular test image for face detection.)