Introduction
This blog covers the second part of Orchestrating Application Workloads in Distributed Embedded Systems. We will go over how to expose Greengrass IPC in a Nomad cluster and have containerized applications publishing metrics to AWS IoT Core using the Greengrass IPC.
Prerequisites
It is essential to follow the first part of the blog, which covers bootstrapping devices with AWS IoT Greengrass and HashiCorp Nomad. Once that is done, we can jump into the application part and the required configuration. As a reminder, the source code is located here: https://github.com/aws-iot-builder-tools/greengrass-nomad-demo
Greengrass IPC Proxy
In order for applications to access Greengrass IPC, we need to create a proxy. We will use socat
to forward the ipc.socket
via the network (TCP) and then use socat
on the application side to create an ipc.socket
file. The example can be found under ggv2-nomad-setup/ggv2-proxy/ipc/recipe.yaml
. Here we deploy the Nomad job:
job "ggv2-server-ipc" {
datacenters = ["dc1"]
type = "system"
group "server-ipc-group" {
constraint {
attribute = "\${meta.greengrass_ipc}"
operator = "="
value = "server"
}
network {
port "ipc_socat" {
static = 3307
}
}
service {
name = "ggv2-server-ipc"
port = "ipc_socat"
provider = "nomad"
}
task "server-ipc-task" {
driver = "raw_exec"
config {
command = "socat"
args = [
"TCP-LISTEN:3307,fork,nonblock",
"UNIX-CONNECT:$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,nonblock"
]
}
}
}
}
This job will use the AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT
provided by the Greengrass Component deployment and run a socat
command by connecting to the defined socket and exposing it over TCP on a reserved port 3307. Note that the deployment of this job will have constraints and will only target the devices tagged as greengrass_ipc=server
, as this is intended to be deployed only on a client where Greengrass is running.
To deploy this to our Greengrass device, we will use the same methods from the previous blog post. Which should look something like this:
Start with building and publishing the component by doing gdk build
and gdk publish
, making sure you are in the ggv2-nomad-setup/ggv2-proxy/ipc/
directory.
Additionally, to deploy this to the targets, we will need to add this to a deployment.json
:
"ggv2.nomad.proxy.ipc": {
"componentVersion": "1.0.0",
"runWith": {}
}
respecting the name of the component and the version provided by the GDK.
After that executing the command below will deploy it to our target:
aws greengrassv2 create-deployment \
--cli-input-json file://deployment.json\
--region ${AWS_REGION}
Once the command executes successfully, we will be ready to move forward with our application.
Application Overview
We will have a simple application written in python that publishes information about used memory and CPU and publishes this information using the Greengrass IPC to AWS IoT Core. The topic here is constructed by having NOMAD_SHORT_ALLOC_ID
as a prefix followed by /iot/telemetry.
We will use this info later once we scale the application across the cluster and start receiving messages on multiple MQTT topics.
Here is the Python code for the application:
import json
import time
import os
import awsiot.greengrasscoreipc
import awsiot.greengrasscoreipc.model as model
NOMAD_SHORT_ALLOC_ID = os.getenv('NOMAD_SHORT_ALLOC_ID')
def get_used_mem():
with open('/proc/meminfo', 'r') as f:
for line in f:
if line.startswith('MemTotal:'):
total_mem = int(line.split()[1]) * 1024 # convert to bytes
elif line.startswith('MemAvailable:'):
available_mem = int(line.split()[1]) * 1024 # convert to bytes
break
return total_mem - available_mem
def get_cpu_usage():
with open('/proc/stat', 'r') as f:
line = f.readline()
cpu_time = sum(map(int, line.split()[1:]))
idle_time = int(line.split()[4])
return (cpu_time - idle_time) / cpu_time
if __name__ == '__main__':
ipc_client = awsiot.greengrasscoreipc.connect()
while True:
telemetry_data = {
"timestamp": int(round(time.time() * 1000)),
"used_memory": get_used_mem(),
"cpu_usage": get_cpu_usage()
}
op = ipc_client.new_publish_to_iot_core()
op.activate(model.PublishToIoTCoreRequest(
topic_name=f"{NOMAD_SHORT_ALLOC_ID}/iot/telemetry",
qos=model.QOS.AT_LEAST_ONCE,
payload=json.dumps(telemetry_data).encode(),
))
try:
result = op.get_response().result(timeout=5.0)
print("successfully published message:", result)
except Exception as e:
print("failed to publish message:", e)
time.sleep(5)
The application can be found under examples/nomad/nomad-docker-pub/app.py.
On top of this we will be using a Dokerfile
to containerized. In order for this to work with GDK, we will be using build_system: "custom"
and specify the script for building and publishing the image to ECR:
{
"component": {
"nomad.docker.pub": {
"author": "Nenad Ilic",
"version": "NEXT_PATCH",
"build": {
"build_system": "custom",
"custom_build_command": [
"./build.sh"
]
},
"publish": {
"bucket": "greengrass-component-artifacts",
"region": "eu-west-1"
}
}
},
"gdk_version": "1.1.0"
}
Where build.sh
will look like this:
set -e
AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')
AWS_REGION=$(jq -r '.component | to_entries[0] | .value.publish.region' gdk-config.json)
COMPONENT_NAME=$(jq -r '.component | keys | .[0]' gdk-config.json)
COMPONENT_AUTHOR=$(jq -r '.component | to_entries[0] | .value.author' gdk-config.json)
COMPONENT_NAME_DIR=$(echo $COMPONENT_NAME | tr '.' '-')
rm -rf greengrass-build
mkdir -p greengrass-build/artifacts/$COMPONENT_NAME/NEXT_PATCH
mkdir -p greengrass-build/recipes
cp recipe.yaml greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_NAME}/$COMPONENT_NAME/" greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_AUTHOR}/$COMPONENT_AUTHOR/" greengrass-build/recipes/recipe.yaml
sed -i "s/{AWS_ACCOUNT_ID}/$AWS_ACCOUNT_ID/" greengrass-build/recipes/recipe.yaml
sed -i "s/{AWS_REGION}/$AWS_REGION/" greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_NAME_DIR}/$COMPONENT_NAME_DIR/" greengrass-build/recipes/recipe.yaml
docker build -t $COMPONENT_NAME_DIR .
docker tag $COMPONENT_NAME_DIR:latest $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$COMPONENT_NAME_DIR:latest
if aws ecr describe-repositories --region $AWS_REGION --repository-names $COMPONENT_NAME_DIR > /dev/null 2>&1
then
echo "Repository $COMPONENT_NAME_DIR already exists."
else
# Create the repository if it does not exist
aws ecr create-repository --region $AWS_REGION --repository-name $COMPONENT_NAME_DIR
echo "Repository $COMPONENT_NAME_DIR created."
fi
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$COMPONENT_NAME_DIR:latest
The script assumes the AWS CLI is installed and gets all the necessary configuration from the gdk-config.json
. The build script will create the appropriate recipe, build the docker image, login to ECR and push it referencing the component name set in the gdk-config.json
.
Finally the Nomad Job for deploying the application will look like this:
job "nomad-docker-pub-example" {
datacenters = ["dc1"]
type = "service"
group "pub-example-group" {
count = 1
constraint {
attribute = "\${meta.greengrass_ipc}"
operator = "="
value = "client"
}
task "pub-example-task" {
driver = "docker"
config {
image = "{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/{COMPONENT_NAME_DIR}:latest"
command = "/bin/bash"
args = ["-c", "socat UNIX-LISTEN:\$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,fork,nonblock TCP-CONNECT:\$GGV2_SERVER_IPC_ADDRESS,nonblock & python3 -u /pyfiles/app.py "]
}
env {
AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT = "/tmp/ipc.socket"
SVCUID="$SVCUID"
}
template {
data = <<EOF
# Get all services and add them to env variables with their names
{{ range nomadServices }}
{{- range nomadService .Name }}
{{ .Name | toUpper | replaceAll "-" "_" }}_ADDRESS={{ .Address}}:{{ .Port }}{{- end }}
{{ end -}}
EOF
destination = "local/env"
env = true
}
}
}
}
- Constraint -
We are starting with our constraint where this application should be deployed. In this scenario, it would be targeting only Nomad clients where
greengrass_ipc=client
. - Task -
Next, we have our task with the Docker driver. Here, we get the image from the ECR, where the variables
AWS_ACCOUNT_ID
,AWS_ACCOUNT_ID
, andCOMPONENT_NAME_DIR
will be replaced by the build script with the appropriate values. Finally, we come to ourcommand
andargs
. These values would override what is already defined by theDockerfile
. In this scenario, we first create theipc.socket
required by the application usingsocat
. TheSVCUID
will be then provided by the Greengrass component at the time of running the job, thus provided as an environment variable inside the Docker container. - Template -
After that, we have a template section that we require to obtain the IP address of our
ggv2-server-ipc
service that we created earlier. We do this by listing all the services and getting the IP addresses and exporting them as environment variables by also converting their names to uppercase letters and appending_ADDRESS
at the end. This provides our env variableGGV2_SERVER_IPC_ADDRESS
for oursocat
command that then looks like this:
socat UNIX-LISTEN:$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,fork,nonblock TCP-CONNECT:$GGV2_SERVER_IPC_ADDRESS,nonblock
Which provides the ipc.socket
before running the app:
python3 -u /pyfiles/app.py
Once we have this set, we can then go build and publish the component by doing gdk build
and gdk publish
.
Additionally in order to deploy this to the targets we will need to add this to a deployment.json
:
"nomad.docker.pub": {
"componentVersion": "1.0.0",
"runWith": {}
}
respecting the name of the component and the version provided by the GDK.
After that executing the command below will deploy it to our target:
aws greengrassv2 create-deployment \
--cli-input-json file://deployment.json\
--region ${AWS_REGION}
Now we will be ready to scale our application.
Scaling the Application
As of now we should have our application running and publishing the data from a single client, however if we require to spread this application across the cluster, in this scenario second client, and have it report the memory and CPU usage, we can do this by simply changing the count=1
to count=2
in our job file:
--- a/examples/nomad/nomad-docker-pub/recipe.yaml
+++ b/examples/nomad/nomad-docker-pub/recipe.yaml
@@ -31,7 +31,7 @@ Manifests:
type = "service"
group "pub-example-group" {
- count = 1
+ count = 2
constraint {
attribute = "\${meta.greengrass_ipc}"
operator = "="
And use the same method to redeploy.
Now if we go to AWS console and under AWS IoT → MQTT test client we can subscribe to topics <NOMAD_SHORT_ALLOC_ID>/iot/telemetry
and should be able to see messages coming. In order to get this ID we can simply run the following command on our device where the nomad server is running:
nomad status nomad-docker-pub-example
And we will find the ID int the Allocations section that looks something like this:
Allocations
ID Node ID Task Group Version Desired Status Created Modified
5dce9e1a 6ad1a15c pub-example-group 10 run running 1m28s ago 32s ago
8c517389 53c96910 pub-example-group 10 run running 1m28s ago 49s ago
This will then allow us to construct those MQTT topics and start seeing the messages coming from those two instances of our application:
In the next part we will take a look on how to access AWS services from the device using Token Exchange Service (TES).
Conclusion
In this blog post, we covered how to expose the Greengrass IPC in a Nomad cluster, allowing a containerized application to publish metrics to AWS IoT Core using Greengrass IPC. We demonstrated how to create a proxy using socat
to forward the ipc.socket
via network (TCP) and how to set up an application that reports memory and CPU usage. We also showed how to scale the application across multiple clients in the cluster.
By using AWS IoT Greengrass and HashiCorp Nomad, you can effectively manage, scale and monitor distributed embedded systems, making it easier to deploy and maintain complex IoT applications.
If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on Twitter or LinkedIn.
Top comments (0)