OpenCV Face Detection: Visualized from Adam Harvey on Vimeo.

This video visualizes the detection process of OpenCV’s face detector. The algorithm utilizes the Viola Jones method of calculating the integral image and then performing some calculations on all the areas defined by the black and white rectangles. The sub-window (in red) is scanned across the image at various scales to detect if there is a potential face. In the post-processing stage all the potential faces are checked for overlaps. Typically, 2 or 3 overlapping rectangles are required to confirm a face. Loner rectangles are rejected as false-positives.
This visualization was done as part of the documentation for CV Dazzle, camouflage from face detection. For more information, visit http://cvdazzle.com

CV Dazzle vs OpenCV Face Detection: The Results

In this image the dotted lines represent all possible face detections, including false positives, before post-processing. Solid lines indicate a positive face detection. Each of the different colors represent the different Haar Cascade face profiles that were used. Images without any solid lines are classified as false-positives and have beaten the face detector.

Post-processing in OpenCV helps remove false positives. Though false positives are still possible, they are reduced by searching for overlapping detections. Only detections that overlap at least 3 (default) other rectangles are considered to be a face. The default setting can be changed to 2 for a higher detection rate, but with more false positives. Setting the overlap minimum to 0 returns all possible detections, as shown here by the dotted lines.

The solid lines represent the faces. They are confirmed by multiple overlapping rectangles. And the colors correspond to the difference detection profiles. OpenCV includes four robust profiles for detecting faces in a frontal view. These profiles include the “frontalface_default.xml”, “frontalface_alt.xml”, “frontalface_alt2.xml”, and “frontalface_alt_tree.xml” files. Each detector returns different noticeably different results.

OpenCV is a computer vision framework available in many different programming languages. For this test, I used the Java port created for Processing from http://ubaa.net/shared/processing/opencv/opencv_image.html and then confirmed the results using the standard framework in C.

The images on the bottom row without any solid lines indicate that no faces were found. Although some potential faces were found. They were ignored, because there was not enough confidence.

These images show a proof of concept hair + makeup pattern capable of deceiving the OpenCV face detector. It was developed as my master’s thesis project at the Interactive Telecommunication Program at NYU. A more detailed report will be posted on CVDazzle.com when it’s ready.

Model: Jen Jaffe, Hair: Pia Vivas, Makeup: Leigh Brown

Open source app for visualizing Haar cascade files used for object detection

Download: Mac version | PC Version (both include application and source code)

The Haar cascade visualizer takes input from the OpenCV (type_id=”opencv-haar-classifier”) formatted XML files used for object detection and outputs each stage of the cascade as a separate TIF file. A set of cascade files ships with OpenCV, which can be download from WillowGarage

Features

• Outputs all stages as .tif files
• Option for creating preview of 1st feature for each stage (much faster)
• Runs on Mac, should run on Windows (not tested yet)

Edit Source Code To:

• Change margins, max width, and images per row
• Change output type from TIF to JPG
• Change background image

Notes

• Feel free to use for any project
• Email if you like it or have a great cascade to share
• NB: Some cascades have stages with a lot of features. If you’re using a large cascade be prepared for high CPU usage.
• Built with Processing

CV Dazzle is camouflage from face detection. It’s based on the original Dazzle camo from WWI and targets automated face detection and recognition systems by altering the contrast and spatial relationship of key facial features. For more info, http://cvdazzle.com

Thursday May 6 at 5:20PM

The last 20 minutes of my two years at ITP will be spent sharing what I’ve learned about CV Dazzle and privacy enhancing technologies. Please join me for the public presentation tomorrow at 721 Broadway Floor 4. If you can’t make it, there is a live stream available here http://itp.nyu.edu/thesis/audience.html. For a full lineup of all thesis presentations, there is a schedule posted here http://itp.nyu.edu/shows/thesis2010/.

Preliminary makeup patterns to hide from face detection

About this image:

• Images with a red square tested positive, a face was found
• Images without a red square tested negative, no face was found
• Images under the section “TEST PATTERNS” are made according to results of the Haar deconstruction
• Images under “RANDOM PATTERNS” are random doodles made without the anti-face detection patterns in mind
• Images underneath the “NO PATTERNS” heading are left untouched to show that the face detection works well on simple line drawings
• Line drawings are from Figure Drawing for Fashion Design

CV Camouflage Test Patterns

What would camouflage look like if it let you hide from a camera?

By reverse engineering the algorithms behind face detection, I generated a preliminary series of images that could be the building blocks of anti-surveillance makeup. The image was made from a single Haar cascade file. Each square represents one “tree” in the cascade file. You can see that the first and second “trees” are the simplest. It’s hard to interpret much beyond here with the human eye, though some images appear to contain face-like representations when you squint at them. If the cascade files are built in order of important, with the first tree being the most important test to pass, then the makeup pattern I’m creating would need to target the images on the first row and it could ignore the rest of the more complicated patterns.

Test patterns based on reversing Haar cascade files.

In the test images above, the original face was overlaid with hand-made variations of the true Haar cascade points of interest (POI) and the reversed POI. Variations of original POI yielded nearly 100% positive results except for the all white overlay (top middle). It was surprising the rest of the all-black/white pixels still tested positive, so one missed face was not too disappointing. All of the expected negatives tested negative. (Images in red are positive faces. Unaltered images were negative)

The images with a light overlay have white pixels where dark pixels should be if there, statistically, was face. Images with a dark overlay have black pixels where the light pixels should be. Because they are all custom variations of the original patterns (top 2 rows), they don’t all hide the face. But the patterns that do hide the images are surprisingly small. Note the small white accents that yielded a negative result and the large black pixel-beard that tested positive.

haarcascade_frontalface_default

Visualizing the features that Haar-based face detection programs use to find faces.

When you use face detection in Processing and OpenFrameworks you’re using an algorithm developed in 2001 called the Viola-Jones method. Though seemingly old, it’s still a robust and efficient way to detect faces in real time. It works great in Java, C++ and even Flash as you can see from these examples in Processing, OpenFrameworks, and Flash.

The algorithm looks at the image (or video) and tries to match the Haar features (in black and white) to features of the image. The more features match, the more likely it is that the image contains a face. As its name implies, a cascade is made of a series of feature-tests. For a face to be detected, the algorithm must match a cascade or series of features in the expected locations. The examples above and below show the Haar features from the cascade files that ship with OpenCV. For more information on how face detection works, I recommend starting with a 2007 article from Servo magazine about how face detection works.

To do: Find the order and weight of the trees in the cascade to determine which are the most and least important features. (NB: the image of Lena is popular test image for face detection.)