Building & Deploying an Image Classification Web App with GCP AutoML Vision Edge, Tensorflow.js & GCP App Engine

In this article, we’ll be learning how we can make a custom image classification web application with a minimal amount of code using AutoML & deploy it on Google Cloud App Engine.

For this article, I will be building a model which will classify whether a person is wearing a face mask or not.

You can check my project by visiting this link on a laptop/PC with a webcam, ensure that your face is well lit for better predictions.

By the end of this article, you will be able to build your image classification model just by changing the dataset.

What is AutoML Vision?

AutoML allows us to train our custom model with our data. It uses NAS(Neural Architecture Search) to find the best way to train our models. The only thing we need to do is to gather data, to enhance the accuracy of the model. It exponentially reduces the effort of writing code to train a model from scratch.

Steps Involved

1. Set-up & Configure a Google Cloud Project
2. Set-up AutoML Vision
3. Creating a Dataset
4. Training The Model
5. Evaluate The Model
6. Testing The Model
7. Exporting the Model as a TensorFlow.js model
8. Building the Web Application
9. Hosting the Application on App Engine

Want to skip to these steps and jump over to the code? It’s all available in my GitHub repository at:

1. Set-up & Configure a Google Cloud Project

Head over to Google Cloud Console, log in with your Google account and create a new project/ choose an existing project by following this link. Next up you need to set up a billing account for this project. Learn how to confirm billing is enabled for your project.

If you are a student, you can apply for free credits worth $50/year, access to free hands-on training and more at edu.google.com without submitting any credit card details.

2. Set-up AutoML Vision

Now let’s enable the Cloud AutoML API by following steps.
You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation Menu at the top-left.
From the Navigation menu select APIs & Services > Library.

export PROJECT_ID=$DEVSHELL_PROJECT_ID
export USERNAME=<USERNAME>

Now run the following command to give AutoML admin permissions to your account:

gcloud projects add-iam-policy-binding $PROJECT_ID \ 
    --member="user:$USERNAME" \
    --role="roles/automl.admin"

Next, we’ll create a Cloud Storage Bucket in the region us-central1, to store our images using the following command(alternatively, you can do the same using the UI by going to Navigation Menu > Storage > Browser > Create Bucket) :

gsutil mb -p $PROJECT_ID \ 
   -c standard    \
   -l us-central1 \
   gs://$PROJECT_ID-vcm/

3. Creating a Dataset

Download and store the images in separate folders according to their labels, for eg. I have stored the images of people wearing a mask in a directory named ‘mask’ and images with people without a mask in a directory named ‘No Mask’ and compressed those folders for upload.

Now, let’s head over to the AutoML UI in a new browser window & click on New Dataset.

4. Training The Model

Once you are satisfied with importing & labelling the Dataset, proceed to the TRAIN tab and click START TRAINING.
Now you’ll be asked to define your model. Give a name to your model and select Edge for creating a downloadable model. Click CONTINUE.
Now you need to specify whether you want the model to be fast or accurate. To find a balance between these select the Best trade-off option & click CONTINUE.
Then you’ll be asked to set a node hour budget for training your model. The more training hours you give, the higher accuracy will be. But the documentation also tells us that if the model stops improving, the training will stop. It’s a good practice to choose the recommended node-hour budget.

While your model is training you can close the tab and maybe relax a bit by having a cup of coffee ☕. Once training is completed, you’ll get an email notification.

5. Evaluate Your Model

Once you receive the email notification you can head over to Evaluating your model, but before we evaluate the model, we should understand the meanings of both precision and recall. Precision refers to the percentage of your results that are relevant. On the other hand, recall refers to the percentage of the total relevant results correctly classified by your algorithm. You can learn more about precision and recall here.

You can see many new terminologies, about which you can learn more here.

6. Testing The Model

Under the TEST & USE tab, click on UPLOAD IMAGES button to upload an image and generate predictions. You can upload 10 images at a time.

Congratulations, you’ve successfully created an image classification model. But wait, it isn’t available for other users to interact with.

7. Exporting the Model as a TensorFlow.js model

Before we proceed further, we would need gsutil : a Python application that lets us access Cloud Storage from the command line.

If you have PyPI (Python Package Index) installed, you can run pip install gsutil or you can Install gsutil as part of the Google Cloud SDK.
Under the TEST & USE tab, you can see many different options for exporting and using your model, we’ll use the TensorFlow.js model.

TensorFlow.js is a library that helps to use ML directly in the browser. AutoML takes very little time to create a model and TensorFlow.js is the easiest and the most efficient way to run models directly inside the browser.

gsutil cp gs://<cloud-storage-bucket>/model-export/icn/* <local-folder>
for example
gsutil cp gs://mask-detection-pbl-vcm/model-export/icn/* Downloads

Once the model is downloaded, you’ll see a model.json file, which contains the tensor information along with the weight file names, .bin files containing the model weights & a dict.txt file, containing the labels which in my case is Mask & No Mask.

. 8.Writing The Web Application

8.1 Creating the index.html file

In the same folder, create an index.html file and copy the following code:

<html>
<head>
<script src="https://unpkg.com/@tensorflow/tfjs"></script>
<script src="https://unpkg.com/@tensorflow/tfjs-automl"></script>
</head>
<body >

<video autoplay playsinline muted id="webcam" width="224" height="224"> 
</video>

<div id="predictions-mask"></div>
<div id="predictions-no-mask"></div>

<button type="button" id="startPredicting" onclick="startPredicting()"> Start Predicting </button>
<button type="button" id="stopPredicting" onclick="stopPredicting()" >Stop Predicting</button>
<script src= "index.js"></script>
</center>
</body>
</html>

The tfjs-automl and tfjs scripts contain the functions required to run the model. If you want to use the model offline, you can download a copy of these scripts and include them in your html file.
The <video> tag creates a video element on the webpage. The 2 <div> will be used to write the predictions from the model. The buttons will be used to Start & Stop predictions, respectively. The script <index.js> is where we will implement the model.

8.2 Creating the index.js file

Now, we’ll create a <index.js> file in the same folder and copy the following code:

const webcamElement= document.getElementById('webcam');
let net;
let isPredicting = false;
function startPredicting(){
 isPredicting=true;
 app();
}
function stopPredicting(){
 isPredicting=false;
 app();
}
async function app(){
 console.log('Loading model..');
 net= await tf.automl.loadImageClassification('model.json');
 console.log('Successfully loaded model');

 const webcam = await tf.data.webcam(webcamElement);
 while(isPredicting){
 const img = await webcam.capture();
 const result = await net.classify(img);

 console.log(result);

 document.getElementById("predictions-mask").innerText=result['0']['label']+": "+Math.round(result['0']['prob']*100)+"%";
 document.getElementById("predictions-no-mask").innerText=result['1']['label']+": "+Math.round(result['1']['prob']*100)+"%";
img.dispose();

await tf.nextFrame();

 }

}

You may get overwhelmed by looking at this code so let me explain it for you. Here, inside the asynchronous function app() , the tf.automl.loadImageClassification() loads the model model.json for us and stores it as net.
tf.data.webcam() will set up the webcam. webcam.capture() will capture images from the live input through webcam and store it as img.
We then pass this image to the model using net.classify(img) and the predictions are stored in the variable result.
The functions startPredicting() & stopPredicting() act as switches to trigger an infinite loop for image classification.
Now you might wonder what these lines do:

document.getElementById("predictions-mask").innerText=result['0']['label']+": "+Math.round(result['0']['prob']*100)+"%";
 document.getElementById("predictions-no-mask").innerText=result['1']['label']+": "+Math.round(result['1']['prob']*100)+"%";

The output that we get is stored in result in this form

[0: {label: “Mask”, prob: 0.048721060156822205}, 1: {label: “No Mask”, prob: 0.9479466080665588}]

The above code is used to write this result on the HTML page as Mask: 5% & No Mask: 95%.
Lastly, img.dispose() disposes of the tensor to release the memory & tf.nextFrame() gives some breathing room by waiting for the next animation frame to fire.

To run the web app download & launch Web Server for Chrome & set-up a by choosing the directory where the code is located.

The app should look like this:

Feel free to add some CSS to the web-page to make it look fancier.

9. Hosting the application on App Engine

Login to your Google Cloud Console and launch the cloud shell and open the Editor.
Inside the editor create a directory, let’s call it www & upload all your files related to the model and web page inside this directory.
In the root directory (outside www ) create a file app.yaml & paste the following code in it:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /
  static_files: www/index.html
  upload: www/index.html

- url: /(.*)
  static_files: www/\1
  upload: www/(.*)

The app.yaml file is a configuration file that tells App Engine how to map URLs to your static files. In the following steps, you will add handlers that will load www/index.html when someone visits your website, and all static files will be stored in and called from the www directory.

Now switch to the cloud shell and run the following command to deploy the web app:

gcloud app deploy

To launch your browser and view the app at https://PROJECT_ID.REGION_ID.r.appspot.com, run the following command:

gcloud app browse

Conclusion

With this, we’ve successfully built and deployed an image classification web app on Google Cloud. Thank you so much for reading patiently. 😃