DEV Community

Cover image for Finger Detection and Tracking using OpenCV and Python
Amar Prakash Pandey
Amar Prakash Pandey

Posted on • Updated on

Finger Detection and Tracking using OpenCV and Python

TL;DR. Code is here.

Finger detection is an important feature of many computer vision applications. In this application, A histogram based approach is used to separate out the hand from the background frame. Thresholding and Filtering techniques are used for background cancellation to obtain optimum results.

One of the challenges that I faced in detecting fingers is differentiating a hand from the background and identifying the tip of a finger. I’ll show you my technique for tracking a finger, which I used in this project. To see finger detection and tracking in action check out this video.

In an application where you want to track a user’s hand movement, skin color histogram will be very useful. This histogram is then used to subtracts the background from an image, only leaving parts of the image that contain skin tone.

A much simpler method to detect skin would be to find pixels that are in a certain RGB or HSV range. If you want to know more about this approach follow here.

The problem with the above approach is that changing light conditions and skin colors can really mess with the skin detection. While on the other hand, Histogram tends to be more accurate and takes into account the current light conditions.

Hand over the rectangles

Green rectangles are drawn on the frame and the user places their hand inside these rectangles. Application is taking skin color samples from the user’s hand and then creates a histogram.

The rectangles are drawn with the following function:

There’s nothing too complicated going on here. I have created four arrays hand_rect_one_x, hand_rect_one_y, hand_rect_two_x, hand_rect_two_y to hold the coordinates of each rectangle. The code then iterates over these arrays and draws them on the frame using cv2.rectangle. Here total_rectangle is just the length of the array i.e. 9.

Now that the user understands where to place his or her palm, the succeeding step is to extract pixels from these rectangles and use them to generate an HSV histogram.

Here function transforms the input frame to HSV. Using Numpy, we create an image of size [90 * 10] with 3 color channels and we name it as ROI (Region of Intrest). It then takes the 900-pixel values from the green rectangles and puts them in the ROI matrix.

The cv2.calcHist creates a histogram using the ROI matrix for the skin color and cv2.normalize normalizes this matrix using the norm Type cv2.NORM_MINMAX. Now we have a histogram to detect skin regions in the frames.

Now that the user understands where to place his or her palm, the succeeding step is to extract pixels from these rectangles and use them to generate an HSV histogram.

Now that we hold a skin color histogram we can use it to find the components of the frame that contains skin. OpenCV provides us with a convenient method, cv2.calcBackProject, that uses a histogram to separate features in an image. I used this function to apply the skin color histogram to a frame. If you want to read more about back projection, you can read from here and here.

In the first two lines, I changed the input frame to HSV and then applied cv2.calcBackProject with the skin color histogram hist. Following that, I have used Filtering and Thresholding function to smoothen the image. Lastly, I masked the input frame using the cv2.bitwise_and function. This final frame should just contain skin color regions of the frame.

Hand seperated from background (1)

Hand seperated from background (2)

Now we have a frame with skin color regions only, but what we really want is to find the location of a fingertip. Using OpenCV you can find contours in a frame if you don’t know what contour is you can read here. Using contours you can find convexity defects, which will be potential fingertip location.

In my application, I needed to find the tip of a finger with which a user is aiming. To do this I determined the convexity defect, which is furthest from the centroid of the contour. This is done by the following code:

Contour in Frame (1)

Contour in Frame (2)<br>

Then it determines the largest contour. For the largest contour, it finds the hull, centroid, and defects.

Defects in red circle and Centroid in purple circle

Now that you have all these defects you find the one that is farthest from the center of the contour. This point is assumed to be the pointing finger. The center is purple and the farthest point is red. And there you have it, you’ve found a fingertip.

Centroid in purple color and Farthest point in red color

All hard part is done up until now, now all we have to do is to create a list to store the changed location of the farthest_point in the frame. It’s up to you that how many changed points you want to store. I am storing only 20 points.

Finger Detection and Tracking using OpenCV and Python<br>

Lastly, thank you for reading this post. For more awesome posts, you can also follow me on Twitter — iamarpandey, Github — amarlearning.

Happy coding! 🤓

Top comments (23)

Collapse
 
tungts1101 profile image
tungts1101

Got this error after I pressed 'z' key:

Traceback (most recent call last):
File "FingerDetection.py", line 186, in
main()
File "FingerDetection.py", line 171, in main
manage_image_opr(frame, hand_hist)
File "FingerDetection.py", line 136, in manage_image_opr
contour_list = contours(hist_mask_image)
File "FingerDetection.py", line 23, in contours
_, cont, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
ValueError: not enough values to unpack (expected 3, got 2)

Do you have any idea?

Collapse
 
amarlearning profile image
Amar Prakash Pandey

@tungts1101 I will suggest you use opencv-python version 3.4.*. You are facing this issue because you are using some higher version of opencv-python.

You can download the correct dependencies using requirement.txt file present in the repository.

Hint: Use command sudo pip install -r requirement.txt

Collapse
 
tungts1101 profile image
tungts1101

That's work. But I think there's still something need to be done in order to eliminate noise in background.

Thread Thread
 
amarlearning profile image
Amar Prakash Pandey

Yes, that's an issue. You can take that up if you want.

Collapse
 
databits profile image
Mitch

Update this line...
From:
_, cont, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

To:
cont, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

That should fix the issue for CV 4.5

Collapse
 
mrcomnguoi profile image
MrComNguoi

Thank you, it works

Collapse
 
lukasswiss profile image
lukasswiss

Hi, this is a great tutorial but my program will not recognize my hand. I hold it up in the proper position, but the green boxes remain. I know that my code is not wrong because I cross-referenced it with your github repo. Do you have any idea why this could be happening?

Collapse
 
amarlearning profile image
Amar Prakash Pandey • Edited

Hi @lukasswiss , I am glad that you liked my post.

Once you place your hand to cover all boxes, you have to press 'Z' key.

Collapse
 
jeisonsanches profile image
Jeison Sanches

Amazing!

Collapse
 
amarlearning profile image
Amar Prakash Pandey • Edited

Thanks, Jeison!

Collapse
 
kataras profile image
Gerasimos (Makis) Maropoulos

Nice Amar, will be interested to see more from you here although I am not even a python coder!

Collapse
 
amarlearning profile image
Amar Prakash Pandey

Thanks @kataras , I am really happy that you liked it. 😊

Collapse
 
talhasharif281 profile image
talhasharif281

Can we do this in MATLAB? If yes, how?

Collapse
 
amarlearning profile image
Amar Prakash Pandey

I have no idea about MATLAB, so can't say anything.

Collapse
 
ben profile image
Ben Halpern

Wow this is super cool stuff!

Collapse
 
amarlearning profile image
Amar Prakash Pandey • Edited

Thanks @ben , glad you liked it!

Collapse
 
lixingyu1591 profile image
lixingyu1591

It's awesome man! However, could you please tell me what kind of algorithm you used in calculating the farthest point?

Collapse
 
geeva22 profile image
Geeva

What is the use of this project? Problem statement?

Collapse
 
martinneumann profile image
Info Comment hidden by post author - thread only accessible via permalink
Martin Neumann

You copied this verbatim from here: benmeline.com/finger-tracking-with...

Please name your sources at least!

Collapse
 
subinsaju profile image
SUBIN SAJU

sir can we do if we write 5 in air then the output as 5 will be in the output can we do like that

Collapse
 
amarlearning profile image
Amar Prakash Pandey

Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more