DEV Community

Viper
Viper

Posted on

Day4: Playing Trex Game on Chrome using Image Processing

#python #machinelearning

Hello everyone, in the last part, we used gestures to perform jump and duck but in this part, we will not use gestures. We will use only some image processing techniques and then let the code play it. But our code will not be any kind of Machine Learning code or algorithms.

This blog is the part of the series #7DaysOfComputerVisionProjects. Links to the blogs and videos of each projects are:

  1. Real-time Background Changing: Video | Blog
  2. Air Mouse: Control Mouse with Gestures Video | Blog
  3. Play Trex Game With Gesture Video | Blog
  4. Auto Dino: Play Trex Game Automatically Video | Blog
  5. Gesture Based Writing Video | Blog
  6. Game: Kill The Fly Video | Blog
  7. Gesture Based Calculator Video | Blog

Preliminary Actions

Import Libraries

We have used everything imported in this code here except pyautogui please install it using pip install pyautogui. We will use it for getting the screenshot of our screen.

import pyautogui
import numpy as np
import tkinter as tk
import cv2
import matplotlib.pyplot as plt
import keyboard
Enter fullscreen mode Exit fullscreen mode

Show Function 

def show(img, fsize=(10,10)):
    figure=plt.figure(figsize=fsize)
    plt.imshow(img)
    plt.show()
show(np.random.randint(0, 255, (100, 100)))
Enter fullscreen mode Exit fullscreen mode

png

Getting Screen Size 


root = tk.Tk()

screen_width = root.winfo_screenwidth()
screen_height = root.winfo_screenheight()

ssize = (screen_height, screen_width)
ssize
Enter fullscreen mode Exit fullscreen mode 
(768, 1366)
Enter fullscreen mode Exit fullscreen mode

Prepare Basic Templates

In this project, we will do background subtraction for checking if any object is coming ahead. So, first I took a screenshot of my dino game then saved it on my project directory.

img

Then I extracted the portion where dino lies, seems like it lies between row 370 to 500 and column 50 to 180. Another 2 things we need is ROIs for checking bottom part and top part. I named broi for bottom and troi for top. Once cropped these parts, show them.
Enter fullscreen mode Exit fullscreen mode 
game_window = cv2.imread("chrome_dino.png", 0)
show(game_window, (15, 15))


brr1,brr2,brc1,brc2 = 370, 450, 160, 580
trr1,trr2,trc1,trc2 = 170, 370, 160, 600

troi = game_window[trr1:trr2, trc1:trc2]
broi = game_window[brr1:brr2, brc1:brc2]
dino = game_window[370:500, 50:180]
show(dino)
show(broi, fsize=(15, 15))
Enter fullscreen mode Exit fullscreen mode

png

png

png

Image Processing to Play Dino

  • Take the resolution of screen. 
resolution = ssize
Enter fullscreen mode Exit fullscreen mode
  • Start a new OpenCV window and name it live where we will make our screenshot live. Also resize it. 
cv2.namedWindow('live', cv2.WINDOW_NORMAL)
cv2.resizeWindow("live", 480, 270)
Enter fullscreen mode Exit fullscreen mode
  • Start variables fc for frame count and increase_every as threshold to increase ROI in ever that frame. 
fc = 0
increase_every = 7
Enter fullscreen mode Exit fullscreen mode
  • Start a main loop. 
while True:
Enter fullscreen mode Exit fullscreen mode
  • Take screenshot. 
    img = pyautogui.screenshot()
Enter fullscreen mode Exit fullscreen mode
  • Convert image to array and then to Grayscale image. 
    frame = np.array(img)
    gframe = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
Enter fullscreen mode Exit fullscreen mode
  • Take ROIs, frame_roi as ROI from each frame and broi as ROI from default image. 
    frame_roi = gframe[brr1:brr2, brc1:brc2]
    broi = game_window[brr1:brr2, brc1:brc2]
Enter fullscreen mode Exit fullscreen mode
  • Check how much of dino is found in frame. Also define a threshold and get location where threshold matches. Something like a mask. 
    res = cv2.matchTemplate(gframe, dino, cv2.TM_CCOEFF_NORMED)

    th = 0.5
    w, h = dino.shape
    loc = np.where(res>=th)
Enter fullscreen mode Exit fullscreen mode
  • If the length of first value of loc is greater or equal to 1 then we suppose that we are currently in the trex game window. Else we will not play a game. 
    if len(loc[0]>=1):
Enter fullscreen mode Exit fullscreen mode
  • If our frame count is equal to increase every, which means we should increase ROIs now. Then we will make frame count to 0 again and increase right side of ROI by 1 column. Also we will check if it exceeds our max column numbers. We then take ROIs again. 
        if fc==increase_every:
            fc = 0
            brc2 += 1
            print("Increased brc2 to: ", brc2)
            if brc2>resolution[1]:
                brc2=resolution[1]
            broi = game_window[brr1:brr2, brc1:brc2]
            frame_roi = gframe[brr1:brr2, brc1:brc2]
Enter fullscreen mode Exit fullscreen mode
  • Check if the dino is around 20 pixels far from 370 and if it is then we will check if we need to jump or not. The 370 here depends on my screenshot image size and screen resolution. The value 370 depends on the width of the screen. We then take an absolute difference between broi and frame_roi. If the value is true, then we find absolute difference and then apply strict normalization. Which simply tells us that the similar parts will be 0 in the result and dissimilar will be 255. 
        if np.abs(loc[0].max()-370)<=20:
            diff = cv2.absdiff(frame_roi.astype(np.float32), broi.astype(np.float32)).astype(np.uint8)

            diff[diff<50] = 0
            diff[diff>=50] = 255
Enter fullscreen mode Exit fullscreen mode
  • We then check for the number of white pixels. White pixels represents that something is not as usual here. Thus we conclude it as an object. But again, if there are more than 100 white pixels, then we will jump. Because we knew that there is an object ahead. Again 100 depends on the screen size. Then we increase the frame count. Just to see the result, we will view the difference. 
            if (diff==255).sum() > 100:
                keyboard.press_and_release("space")
                fc+=1

            cv2.imshow("diff", diff)
Enter fullscreen mode Exit fullscreen mode
  • Else, which means dino has not been found, we will not do anything but reset the ROI coordinates. 
    else:
        brr1,brr2,brc1,brc2 = 370, 450, 160, 520
Enter fullscreen mode Exit fullscreen mode
  • Finally, show each frames and then wait for escape key. 
    cv2.imshow("live", frame)


    if cv2.waitKey(1) &0xFF == 27:
        break
cv2.destroyAllWindows()
Enter fullscreen mode Exit fullscreen mode

Complete Code 

game_window = cv2.imread("chrome_dino.png", 0)
show(game_window, (15, 15))

brr1,brr2,brc1,brc2 = 370, 450, 160, 520
trr1,trr2,trc1,trc2 = 170, 370, 160, 600

troi = game_window[trr1:trr2, trc1:trc2]
broi = game_window[brr1:brr2, brc1:brc2]
dino = game_window[370:500, 50:180]
# show(dino)
# show(roi, fsize=(15, 15))    


resolution = ssize 
cv2.namedWindow('live', cv2.WINDOW_NORMAL)
cv2.resizeWindow("live", 480, 270)

fc = 0
increase_every = 7

while True:
    img = pyautogui.screenshot()


    frame = np.array(img)
    gframe = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    frame_roi = gframe[brr1:brr2, brc1:brc2]
    broi = game_window[brr1:brr2, brc1:brc2]

    res = cv2.matchTemplate(gframe, dino, cv2.TM_CCOEFF_NORMED)

    th = 0.5
    w, h = dino.shape
    loc = np.where(res>=th)
    #print(loc)
    if len(loc[0]>=1):
        game_stat = "start"
        if fc==increase_every:
            fc = 0
            brc2 += 1
            print("Increased brc2 to: ", brc2)
            if brc2>resolution[1]:
                brc2=resolution[1]
            broi = game_window[brr1:brr2, brc1:brc2]
            frame_roi = gframe[brr1:brr2, brc1:brc2]

        if np.abs(loc[0].max()-370)<=20:
            diff = cv2.absdiff(frame_roi.astype(np.float32), broi.astype(np.float32)).astype(np.uint8)

            diff[diff<50] = 0
            diff[diff>=50] = 255

            if (diff==255).sum() > 100:
                keyboard.press_and_release("space")
                fc+=1

            cv2.imshow("diff", diff)
    else:
        game_state="pause"
        brr1,brr2,brc1,brc2 = 370, 450, 160, 520

    cv2.imshow("live", frame)


    if cv2.waitKey(1) &0xFF == 27:
        break
cv2.destroyAllWindows()
Enter fullscreen mode Exit fullscreen mode

png

Increased brc2 to:  521

Increased brc2 to:  522

Increased brc2 to:  523

Increased brc2 to:  524

Increased brc2 to:  525

Increased brc2 to:  526

Increased brc2 to:  527

Increased brc2 to:  528

Increased brc2 to:  529

Increased brc2 to:  530

Increased brc2 to:  521

Increased brc2 to:  522

Increased brc2 to:  523

Increased brc2 to:  524

Increased brc2 to:  525

Increased brc2 to:  526

Increased brc2 to:  527

Increased brc2 to:  528

Increased brc2 to:  529

Increased brc2 to:  530

Increased brc2 to:  531

Increased brc2 to:  532

Increased brc2 to:  533

Increased brc2 to:  534

Increased brc2 to:  535

Increased brc2 to:  536

Increased brc2 to:  537
Enter fullscreen mode Exit fullscreen mode




Finally

This is all for this part and I have not written a code for ducking part. I hope you can try that feature on your own. If you found some errors or problems please let us know.

Top comments (0)

Read next

ppaanngggg profile image

YOLOv11: A New Breakthrough in Document Layout Analysis

ppaanngggg -

rumca-js profile image

Django bookmark management software

Piotr -

mikeyoung44 profile image

Unlocking Search Relevance: Large Language Models Power Pinterest Discovery

Mike Young -

mikeyoung44 profile image

Breakthrough in AI Memory: Endowing Language Models with Human-like Episodic Recall

Mike Young -