What is a Convolutional Neural Network (CNN)? How can they be used to detect features in images? This is the video of a live coding session in which I show how to build a CNN in Python using Keras and extend the "smile detector" I built last week to use it.
A 1080p version of this video can be found on Cinnamon
A Convolutional Neural Network is a particular type of neural network that is very suited to analysing images. It works by passing a 'kernel' across the input image (convolution) to produce an output. These convolutional layers are stacked to produce a deep learning network and able to learn quite complex features in images.
In this session I coded a simple 3-layer CNN and trained it with manually classified images of faces.
Much of the code was based on the previous iteration of this. Subsequent to the live coding session, I actually refactored the code to use python generators to simplify the processing pipeline.
Frame Generator
This method opens the video file and iterates through the frames yielding each frame.
def frame_generator(self, video_fn):
cap = cv2.VideoCapture(video_fn)
while 1:
# Read each frame of the video
ret, frame = cap.read()
# End of file, so break loop
if not ret:
break
yield frame
cap.release()
Calculating the Threshold
Like in the previous session, we iterate through the frames to calculate the different between each frame and the previous one. It then returns the threshold needed in which to filter out just the top 5% of images:
def calc_threshold(self, frames, q=0.95):
prev_frame = next(frames)
counts = []
for frame in frames:
# Calculate the pixel difference between the current
# frame and the previous one
diff = cv2.absdiff(frame, prev_frame)
non_zero_count = np.count_nonzero(diff)
# Append the count to our list of counts
counts.append(non_zero_count)
prev_frame = frame
return int(np.quantile(counts, q))
Filtering the Image Stream
Another generator that takes in an iterable of the frames and a threshold and then yields each frame whose difference from the previous frame is above the supplied threshold.
def filter_frames(self, frames, threshold):
prev_frame = next(frames)
for frame in frames:
# Calculate the pixel difference between the current
# frame and the previous one
diff = cv2.absdiff(frame, prev_frame)
non_zero_count = np.count_nonzero(diff)
if non_zero_count > threshold:
yield frame
prev_frame = frame
Finding the Smiliest Image
By factoring out the methods above we can chain the generators together and pass them in to this method to actually look for the smiliest image. This means that (unlike the previous version) this method doesn't need to concern itself with deciding which frames to analyse.
We use the trained neural network (as a Tensorflow Lite model) to predict whether a face is smiling. Much of this structure is similar to last session in which we first scan the image to find faces. We then align each of those faces using a facial aligner -- this transforms the face such that the eyes are in the same location of each image. We pass each face into the neural network that gives us a score from 0 to 1.0 of how likely it is smiling. We sum all those values up in order to get an overall score of 'smiliness' for the frame.
def find_smiliest_frame(self, frames, callback=None):
# Allocate the tensors for Tensorflow lite
self.interpreter.allocate_tensors()
input_details = self.interpreter.get_input_details()
output_details = self.interpreter.get_output_details()
def detect(gray, frame):
# detect faces within the greyscale version of the frame
faces = self.detector(gray, 2)
smile_score = 0
# For each face we find...
for rect in faces:
(x, y, w, h) = rect_to_bb(rect)
face_orig = imutils.resize(frame[y:y + h, x:x + w], width=256)
# Align the face
face_aligned = self.face_aligner.align(frame, gray, rect)
# Resize the face to the size our neural network expects
face_aligned = face_aligned.reshape(1, 256, 256, 3)
# Scale to pixel values to 0..1
face_aligned = face_aligned.astype(np.float32) / 255.0
# Pass the face into the input tensor for the network
self.interpreter.set_tensor(input_details[0]['index'],
face_aligned)
# Actually run the neural network
self.interpreter.invoke()
# Extract the prediction from the output tensor
pred = self.interpreter.get_tensor(
output_details[0]['index'])[0][0]
# Keep a sum of all 'smiliness' scores
smile_score += pred
return smile_score, frame
best_smile_score = 0
best_frame = next(frames)
for frame in frames:
# Convert the frame to grayscale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Call the detector function
smile_score, frame = detect(gray, frame)
# Check if we have more smiles in this frame
# than out "best" frame
if smile_score > best_smile_score:
best_smile_score = smile_score
best_frame = frame
if callback is not None:
callback(best_frame, best_smile_score)
return best_smile_score, best_frame
We can then chain the functions together:
smiler = Smiler(landmarks_path, model_path)
fg = smiler.frame_generator(args.video_fn)
threshold = smiler.calc_threshold(fg, args.quantile)
fg = smiler.frame_generator(args.video_fn)
ffg = smiler.filter_frames(fg, threshold)
smile_score, image = smiler.find_smiliest_frame(ffg)
Output
Testing it out it all works pretty well, and finds a nice snapshot from the video of smiling faces.
The full code to this is now wrapped up as a complete Python package:
Smiler
This is a library and CLI tool to extract the "smiliest" of frame from a video of people.
It was developed as part of Choirless as part of IBM Call for code.
Installation
% pip install choirless_smiler
Usage
Simple usage:
% smiler video.mp4 snapshot.jpg
It will do a pre-scan to determine the 5% most changed frames from their previous frame in order to just consider them. If you know the threshold of change you want to use you can use that. e.g.
The first time smiler runs it will download facial landmark data and store it in ~/.smiler
location of this data and cache directory can be specified as arguments
% smiler video.mp4 snapshot.jpg --threshold 480000
Help
% smiler -h
usage: smiler [-h] [--verbose] [--threshold THRESHOLD]
[--landmarks-url LANDMARKS_URL] [--cache-dir CACHE_DIR]
[--quantile QUANTILE]
video_fn image_fn
Save thumbnail of smiliest frame in video
positional arguments:
video_fn filename for video toI hope you enjoyed the video, if you want to catch them live, I stream each week at 2pm UK time on the IBM Developer Twitch channel:
Top comments (1)
Very nice project :)