In this post we will introduce the Azure IoT Edge Camera Tagging Module. This module will deploy a service onto compatible IoT devices to capture images from live RTSP video streams. These images can then be uploaded from the module into Microsoft Cognitive Services for use in training object detection and image classification models, or into Azure Blob Storage for archival purposes.

Introduction

The combination of Artificial Intelligence and Internet of Things yields a paradigm commonly referred to as the "Artificial Intelligence of Things" or AIoT. Development in this field is focused around augmentation of common sensors like microphones and cameras which take their traditional output and use them as algorithmic inputs that are processed by AI models. This can enable abilities that include transforming spoken words into contextual requests that can be operated on by a computer program or determining the presence and location of objects in a video feed.

The development of models for use in AIoT applications often involves gathering quantities of training data to use as inputs that an AI model can learn from. As you might expect, the quality and relevance of the data involved in the training of these models can have a large impact on their accuracy in production. There are a number of ways to approach this, for example using vast training sets that consist of millions or even billions of samples to produce a generic detector for a variety of unique inputs. Another method might involve using contextually relevant data, for example taking samples at the site of deployment and using those to create a tightly tuned model that can handle a specific environment.

The Azure IoT Edge Camera Tagging Module can assist in both of these strategies by allowing you to capture sample data for vision based AI models at scale. It can also allow you to capture data remotely from the site of deployment. This enables solution builders to produce varied and precise AI models using data gathered from a module running on any IoT Edge capable device.

The content will assume some prior knowledge of Azure IoT Edge and it is suggested that you are familiar with installing the IoT Edge Runtime on Linux devices and how to create a deployment for a Linux-Based IoT Edge Device. If you would like to learn more about Azure IoT Edge and are looking for a good place to learn, I highly recommend the official IoT Edge Documentation and the interactive courses on IoT Edge at Microsoft Learn.

Getting Started

Before we begin, the following is assumed:

• You have an active Azure Subscription
• You have followed steps to install the Azure IoT Edge Runtime to your Linux- based ARM32, ARM64, or AMD64 target device
• You have followed the steps to register your IoT Edge Device to an Azure IoT Hub instance
• You are familiar with the steps to deploy modules to an IoT Edge Device from Visual Studio Code
• You have installed the Azure IoT Edge Extension for Visual Studio Code

The Azure IoT Edge Camera Tagging Module is under active development on Github. A quick perusal of the official README mentions that the module supports ARM32, ARM64, and AMD64 Linux platforms. The project is distributed in a fashion that encourages building the module from source and publishing it to your own private docker registry. This is the preferred approach if you plan to use the module in production, but it does require some additional steps and overhead.

To make test-driving a bit easier, I have published the Camera Tagging module for all three supported architectures in a public DockerHub repo using a Docker manifest to allow you to easily install the module regardless of the target platform with a reference the following image tag: toolboc/camerataggingmodule:latest. This multi-platform image will be referenced in our deployment.template.json.

Create a new IoT Edge Solution and update the deployment.template.json

Start by opening Visual Studio Code and use the shortcut (CTRL+SHIFT+P) and search for the Azure IoT Edge: New IoT Edge Solution task:

Choose an appropriate directory to create your project under, next you will be asked to give your solution a name. It is suggested to name it something like CameraTaggingModuleExample.

After your solution is named, you will be prompted to Select Module Template. Select the Empty Solution option.

This will produce a base project with the minimal scaffolding to produce an IoT Edge deployment.

Copy the content below and replace the current contents of the generated deployment.template.json:

{
  "$schema-template": "1.0.0",
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              "containerRegistry": {
                "username": "$CONTAINER_REGISTRY_USERNAME",
                "password": "$CONTAINER_REGISTRY_PASSWORD",
                "address": "$CONTAINER_REGISTRY_NAME"
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
              "createOptions": {}
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0.9",
              "createOptions": {
                "HostConfig": {
                  "PortBindings": {
                    "5671/tcp": [
                      {
                        "HostPort": "5671"
                      }
                    ],
                    "8883/tcp": [
                      {
                        "HostPort": "8883"
                      }
                    ],
                    "443/tcp": [
                      {
                        "HostPort": "443"
                      }
                    ]
                  }
                }
              }
            }
          }
        },
        "modules": {
          "CameraTaggingModule": {
            "version": "1.0.3",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "toolboc/camerataggingmodule:latest",
              "createOptions": {
                "ExposedPorts": {
                  "3000/tcp": {},
                  "3002/tcp": {},
                  "3003/tcp": {}
                },
                "HostConfig": {
                  "PortBindings": {
                    "3000/tcp": [
                      {
                        "HostPort": "3000"
                      }
                    ],
                    "3002/tcp": [
                      {
                        "HostPort": "3002"
                      }
                    ],
                    "3003/tcp": [
                      {
                        "HostPort": "3003"
                      }
                    ]
                  }
                }
              }
            },
            "env": {
              "RTSP_IP": {
                "value": "wowzaec2demo.streamlock.net"
              },
              "RTSP_PORT": {
                "value": "554"
              },
              "RTSP_PATH": {
                "value": "vod/mp4:BigBuckBunny_115k.mov"
              },
              "REACT_APP_SERVER_PORT": {
                "value": "3003"
              },
              "REACT_APP_WEB_SOCKET_PORT": {
                "value": "3002"
              },
              "REACT_APP_LOCAL_STORAGE_MODULE_NAME": {
                "value": "azureblobstorageoniotedge"
              },
              "REACT_APP_LOCAL_STORAGE_PORT": {
                "value": "11002"
              },
              "REACT_APP_LOCAL_STORAGE_ACCOUNT_NAME": {
                "value": "$LOCAL_STORAGE_ACCOUNT_NAME"
              },
              "REACT_APP_LOCAL_STORAGE_ACCOUNT_KEY": {
                "value": "$LOCAL_STORAGE_ACCOUNT_KEY"
              }
            }
          },
          "azureblobstorageoniotedge": {
            "version": "1.2",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azure-blob-storage:latest",
              "createOptions": {
                "Env":[
                  "LOCAL_STORAGE_ACCOUNT_NAME=$LOCAL_STORAGE_ACCOUNT_NAME",
                  "LOCAL_STORAGE_ACCOUNT_KEY=$LOCAL_STORAGE_ACCOUNT_KEY"
                 ],
                 "HostConfig":{
                   "Binds": ["/data/containerdata:/blobroot"],
                   "PortBindings":{
                     "11002/tcp": [{"HostPort":"11002"}]
                   }
                 }
              }
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "azureblobstorageoniotedgeToIoTHub": "FROM /messages/modules/azureblobstorageoniotedge/outputs/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    },
    "azureblobstorageoniotedge":{
      "properties.desired": {
        "deviceAutoDeleteProperties": {
          "deleteOn": false,
          "retainWhileUploading": true
        },
        "deviceToCloudUploadProperties": {
          "uploadOn": true,
          "uploadOrder": "OldestFirst",
          "cloudStorageConnectionString": "$CLOUD_STORAGE_CONNECTION_STRING",
          "storageContainersForUpload": {
            "$LOCAL_STORAGE_ACCOUNT_NAME": {
              "target": "$DESTINATION_STORAGE_NAME"
            }
          },
          "deleteAfterUpload": true
        }
      }
    }
  }
}

We will use this deployment.template.json in upcoming steps to create a deployment manifest using tooling in Visual Studio Code. Notice that the multi-platform image (toolboc/camerataggingmodule:latest) is referenced in the deployment.template.json specification.

We also configure a default RTSP stream using the Big Buck Bunny RTSP stream from Wowza. This is the most reliable publicly accessible RTSP stream on the entire internet, and is provided as an example (trust me, reliable public RTSP streams are very difficult to find).

There is also an azureblobstorageoniotedge module that is included which can allow us to save captured images locally and replicate them to the cloud using the IoT Edge Blob Storage Module. This module is useful in private networks where RTSP streams may be available, but outside internet access is not. With this module configured, we could deploy an IoT Edge device into the environment to capture images, then retrieve it and publish the images to the cloud when outbound network access is restored to the device. This will be covered in more detail later in the next steps.

Configure the Blob Storage Module

In this step, we will configure the IoT Edge Blob Storage Module which can be used in conjunction with the CameraTaggingModule to store image captures locally and replicate them to the cloud. Technically, this module is optional and the CameraTaggingModule can upload images directly to the cloud or CustomVision.AI without it, but it gives a more robust solution for the end user that can capture and store images without the need for outbound internet access.

Open the project created in the previous step and create a file named .env and supply it with the following contents:

CONTAINER_REGISTRY_NAME=
LOCAL_STORAGE_ACCOUNT_KEY=
LOCAL_STORAGE_ACCOUNT_NAME=camerataggingmodulelocal
DESTINATION_STORAGE_NAME=camerataggingmodulecloud
CLOUD_STORAGE_CONNECTION_STRING=

This file will will store key/value that are used to replace values in deployment.template.json to produce a working deployment manifest. You will notice these entries in the deployment.template.json are preceeded with the '$' symbol. This marks them as tokens for replacement during the generation of the deployment manifest.

For now, we will skip the CONTAINER_REGISTRY_NAME as that is only needed if you are pulling container images from a private repository, since the modules in our deployment are all publicly available, it is not needed at this time.

Produce a value for LOCAL_STORAGE_ACCOUNT_KEY by visiting GeneratePlus. This will generate a random base64 encoded string that will be used to configure a secure connection to the local blob storage instance. You will want to supply the entire result, which should end with two equal signs (==).

LOCAL_STORAGE_ACCOUNT_NAME is best left as-is, but you are welcome to rename it, provided that it follows the format for naming: The field can contain only lowercase letters and numbers and the name must be between 3 and 24 characters.

DESTINATION_STORAGE_NAME is supplied from an asssumed-to-exist blob storage container in the Azure Cloud. You can create this container by performing the following steps:

Navigate to the Azure Marketplace and search for 'blob', then select Storage Account - blob, file, table, queue

Create the Storage Account using settings similar to below (note: the Storage account name must be globally unique)

Select Review + Create => Create to deploy the new Storage Account Resource.

Navigate to your newly deployed Storage Account and select Containers:

Create a new storage container named "camerataggingmodulecloud" as shown below (the name is important as it matches the value in the .env):

CLOUD_STORAGE_CONNECTION_STRING can be obtained by visiting your newly created Storage Account and selecting Settings => Access Keys. Copy the entire contents of the Connection string and supply this as the value.

Your completed .env file should look similar to the following:

CONTAINER_REGISTRY_NAME=
LOCAL_STORAGE_ACCOUNT_KEY=9LkgJa1ApIsISmuUHwonxg==
LOCAL_STORAGE_ACCOUNT_NAME=camerataggingmodulelocal
DESTINATION_STORAGE_NAME=camerataggingmodulecloud
CLOUD_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=camerataggingmodulestore;AccountKey=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000==;EndpointSuffix=core.windows.net

One final and very important step, the Azure IoT Edge Blob Storage module runs as a special user with uid (1100). This is alluded to in the official documentation and requires some configuration on the host in order for the local blob storage to be able to write to the host directory. I found that I had to perform this step when deploying to an AMD64 device but it did not appear to be required for AARCH64 devices, your experience may vary.

We will simply create the expected directory that is specified as a bind mount in our deployment.template.json for **azureblobstorageoniotedge* and assign it owner and group permissions for the user that the IoT Edge Blob Storage module runs as. This will allow the module to write to this directory on the host filesystem.

sudo mkdir /data
sudo mkdir /data/containerdata
sudo chown -R 11000:11000 /data/containerdata
sudo chmod -R 700 /data/containerdata

We are now ready to generate the deployment manifest.

Generate and apply the deployment to the IoT Edge device

In this section, we will use the .env file to produce a deployment manifest and then apply it to our IoT Edge device. This deployment will work as-is across ARM64, ARM32, and AMD64 devices as it will reference platform agnostic image tags for all of the included modules (edgeAgent, edgeHub, CameraTaggingModule, azureblobstorageoniotedge).

Within Visual Studio Code, right-click the deployment.template.json file and select Generate IoT Edge Deployment Manifest

This will create a new folder named config and within it there should be deployment manifest created and namded selected architecture. Remember, while the name may imply that this is a platform-specific deployment, this particular deployment is platform agnostic and will work on ARM64, ARM32, or AMD64 platforms.

Right-Click this file and select Create Deployment for Single Device

You will be prompted to select a device that has been registered to your IoT Hub. Choose the appropriate device and the deployment will be applied. In a few moments, your device will begin running the supplied modules (edgeAgent, edgeHub, CameraTaggingModule, azureblobstorageoniotedge).

Using the Deployed CameraTaggingModule

In this section we will explore features of the CameraTaggingModule and demonstrate how you can use it to capture images for use in training object detection and image classification models at CustomVision.AI.

The CameraTaggingModule is configured to expose a web service on port 3000 of the IoT Edge device. You can interact with it by visiting http://<devicehostname>:30000 or http://<deviceipaddress>:3000 on a compatible web browser. The latest version of Microsoft Edge, Firefox, and Chrome all worked for me.

Upon loading, the interface should present you with a playback of Big Buck Bunny RTSP stream:

We can change the current RTSP stream by selecting Change Camera, then supply a Camera Name and associated RTSP Address and Add Camera. Next, choose the newly added camera from the drop-down and select Save.

Here is an example of changing the RTSP stream to one of my home security cameras:

Now we can begin capturing images by selecting Capture then you can supply it with a tag or tags and name the image:

Once you have collected a variety of images, they can be reviewed in the Images panel:

Uploading Tagged Images to CustomVision.AI

You will need to have an existing project set up at CustomVision.AI. You may refer to the following quickstarts to create an image classification model or object detection model.

• Quickstart: How to build a classifier with Custom Vision
• Quickstart: How to build an object detector with Custom Vision

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Custom Vision and you will be presented with the following configuration screen:

To obtain these values, navigate to your CustomVision.AI project and select the "Gear" icon. This will present you with a screen that looks like the following:

Once these values are entered, you will be presented with a dropdown of your existing projects. Choose the intended project you wish to upload your images to and select Push to Custom Vision. After a while you should see Success.

Head back to your CustomVision.AI project to see your newly uploaded and tagged training images:

Now you see just how it easy it can be to use the CameraTaggingModule to gather images from site deployments. Next, we will show how to leverage the Blob storage features to capture images locally and replicate to the cloud for long-term storage.

Using the Blob Storage features of the CameraTaggingModule

The CameraTaggingModule allows for storing image captures locally, where they can then be replicated to Azure Blob Storage, and also allows for direct upload to Azure Blob Storage (bypassing the need to store locally). As mentioned, this is highly useful in environments that do not allow for transmission of data to the outside world. With this feature, you can capture data from air-gapped environments for archival in the cloud.

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Blob Storage and you will be presented with the following options:

To upload directly to blob storage, select the Push to Blob Storage option. You will be presented with a screen asking to supply a Storage Connection String:

You can obtain the string just as we did in previous steps:

Once supplied, you should see a drop-down under Container Name. Choose the storage container that was created earlier then Push to Blob Store.

You can verify that your images have been uploaded to the cloud by visiting your Storage container in the Azure Portal:

Connecting to the local Blob Storage Instance with Azure Storage Explorer

We will now show how you can connect to the local Blob Storage instance using the Azure Storage Explorer. This will make it very easy to view the state of our local blob storage and easily perform operations on it's data.

First, let's store some image data into the local blob storage.

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Blob Storage and you will be presented with the following options:

To upload directly to the local blob storage, select the Push to Local Storage option.

Next, we will install the Azure Storage Explorer

On my Ubuntu 18.04 machine, I accomplished this with the following:

sudo snap install storage-explorer
snap connect storage-explorer:password-manager-service :password-manager-service

Once installed, you can launch the Storage Explorer with:

storage-explorer

Select the Open Connect Dialog icon that looks like a electrical plug and select Use a Connection String:

Enter camerataggingmodulelocal for the Display name then supply a Connection string in the following format:

DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/camerataggingmodulelocal;AccountName=camerataggingmodulelocal;AccountKey=9LkgJa1ApIsISmuUHwonxg==

If you used a different value for LOCAL_STORAGE_ACCOUNT_KEY in your .env file, be sure to replace it with the appropriate value as shown:

Once you are configured appropriately, select Connect:

Similarly, you can repeat this process and add the Connection String used for CLOUD_STORAGE_CONNECTION_STRING in your .env file to have access to the Azure Cloud Storage Instance we created earlier.

Now you can explore the contents of the local and cloud blob storage containers:

You may notice that the local container is empty, if the IoT Edge device has network connectivity to Microsoft Azure, it will delete uploads based on the default desired properties configuration in deployment.template.json:

    "azureblobstorageoniotedge":{
      "properties.desired": {
        "deviceAutoDeleteProperties": {
          "deleteOn": false,
          "retainWhileUploading": true
        },
        "deviceToCloudUploadProperties": {
          "uploadOn": true,
          "uploadOrder": "OldestFirst",
          "cloudStorageConnectionString": "$CLOUD_STORAGE_CONNECTION_STRING",
          "storageContainersForUpload": {
            "$LOCAL_STORAGE_ACCOUNT_NAME": {
              "target": "$DESTINATION_STORAGE_NAME"
            }
          },
          "deleteAfterUpload": true
        }
      }
    }

For details on how to modify these settings further, refer to the official README in the camerataggingmodule github repository.

Automating the CameraTaggingModule with DirectMethods

Direct Methods are a feature of Azure IoT Hubs that can provide remote access to local methods defined in module code in a secure manner. The CameraTaggingModule exposes methods that include image capture, uploading captured images to CustomVision.AI, pushing captured images to local blob storage, and deleting all captured images. By combining these elements together, we can automate an image capture process across any number of available RTSP feeds.

Here is an example bash script capture.sh that can trigger the execution of the capture method:

#!/bin/bash

#Configuration
iothubName=iothubname
deviceId=deviceId
#Generated with 'az iot hub generate-sas-token -n <iothubName> -du 31536000'
SharedAccessSignature='SharedAccessSignature sr=iothubname.azure-devices.net&sig=x&se=x&skn=iothubowner'

usage(){
  echo "***Camera Tagging Module Capture Script***"
  echo "Usage: ./capture.sh <rtsp_ip> <rtsp_port> <rtsp_path>"
}

capture(){
curl -X POST \
  https://$iothubName.azure-devices.net/twins/$deviceId/modules/CameraTaggingModule/methods?api-version=2018-06-30 \
  -H "Authorization: $SharedAccessSignature" \
  -H 'Content-Type: application/json' \
  -d "{
    \"methodName\": \"capture\",
    \"responseTimeoutInSeconds\": 200,
    \"payload\": {
        \"RTSP_IP\":\"$rtsp_ip\",
        \"RTSP_PORT\":\"$rtsp_port\",
        \"RTSP_PATH\":\"$rtsp_path\",
        \"TAGS\":[\"automatedCaptures\"]
    }
  }"
}


# Arguments
rtsp_ip=$1
rtsp_port=$2
rtsp_path=$3

# Check Arguments
[ "$#" -ne 3 ] && { usage && exit 1; } || capture

And here is an example bash script push.sh that can trigger pushing to the local blob storage:

#!/bin/bash

#Configuration
iothubName=iothubName
deviceId=deviceId
#Generated with 'az iot hub generate-sas-token -n <iothubName> -du 31536000'
SharedAccessSignature='SharedAccessSignature sr=iothubname.azure-devices.net&sig=x&se=x&skn=iothubowner'

curl -X POST \
  https://$iothubName.azure-devices.net/twins/$deviceId/modules/CameraTaggingModule/methods?api-version=2018-06-30 \
  -H "Authorization: $SharedAccessSignature" \
  -H 'Content-Type: application/json' \
  -d '{
    "methodName": "push",
    "responseTimeoutInSeconds": 200,
    "payload": {
        "MODULE_NAME":"azureblobstorageoniotedge",
        "STORAGE_PORT":"11002",
        "ACCOUNT_NAME":"camerataggingmodulelocal",
        "ACCOUNT_KEY":"jukoPNlrFwXR/eELSxryaw==",
        "DELETE":"true"
    }
  }'

By combining these together, you can create a bash script automateCapture.sh to automatically gather images in a loop and store them to the local blob storage:

#!/bin/bash

while true; do
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./push.sh
done

Now, if you've followed along up until this point, you might realize that we've just laid out the process for a system that can capture training samples automatically. This means we can gather samples from various times of day in our natural environment without any manual effort!

What's even more amazing is that the system can work even if outbound network access is cut off to the device! That's right! The azureblobstorageoniotedge module will keep a copy locally until network connectivity is achieved where it will then drop the data into an Azure Cloud Storage container. If the device is configured and networked to a Transparent IoT Edge Gateway, the device can even continue to receive Device Method invocations from the cloud! If you are interested in learning more about this concept, there is an interactive module at Microsoft Learn that walks through the process of configuring an IoT Edge Gateway. This is a prime example of Azure IoT Edge's powerful capabilities for orchestrating workloads at the edge, it can continue to work even when the device it is running on is disconnected from the internet!

Conclusion

We have demonstrated a variety of features present in the Azure IoT Edge Camera Tagging module. With this guide, you should be able to successfully deploy and capture images for use in training custom image classification and object detection models using CustomVision.AI.

As you have seen, this is an extremely powerful tool that can allow you to gather training samples from the actual site of deployment, potentially while a solution is even in production! Hopefully this guide has made it very easy to understand how all of the pieces work together and allow you the freedom to deploy it for your own scenarios.

I am personally excited about the prospect of generating samples using my home security cameras to detect my dog, cat, vehicles and family members. Armed with this knowledge, it becomes possible to create a highly accurate system that can be queried to produce insights like "How long has the car been in the driveway?", "Where is the last place the cat was seen?", and "How many cars drove past the driveway last week?".

As the paradigm of AIoT continues to flourish, it is my expectation that we will begin to see consumer systems that are capable of tailoring computer vision workloads to specific environments. The Azure IoT Edge Camera Tagging module is an excellent tool that makes this process that much closer to a reality. If you are interested in developing computer vision solutions at the Edge, I highly recommend it!

Until next time,

Happy Hacking!

-Paul